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Inconsistent  findings  comparing  the  relative  efficacy  of  problem-based  (PBL) 

versus  lecture-based  (LBL)  learning  in  medical  education  motivated  this  study.  In  this 

meta-analysis  of  82  studies  involving  45  universities  in  twelve  countries,  with  12,979 

participants  and  121  findings,  PBL  was  associated  with  a  significant  positive  effect  size 

for  clinical  science  achievement,  problem-solving  achievement,  self-directed  learning 

skills,  and  attitude  toward  learning.  Instructional  effects  were  statistically  equivalent  for 

biomedical  science  achievement.  However,  variance  analysis  revealed  a  subgroup 

exhibiting  a  small,  but  significant  posifive  effect  size  when  biomedical  science 

achievement  was  compared  using  criterion-referenced  rather  than  standardized 

assessments.  Findings  suggest  that  when  achievement  and  problem  solving  ability  are 

examined  with  assessments  contextually  framed  and  requiring  the  integration  and 

application  of  biomedical  and  clinical  science  knowledge,  medical  students  and 

physicians  who  undergo  PBL  significantly  outperform  counterparts  who  undergo  LBL. 


Moreover,  given  the  dominant  factual  and  abstract  nature  of  the  content,  the  lack  of 
unique  superiority  of  LBL  for  biomedical  science  achievement  is  an  important  practical 
finding.  Indeed,  differential  instruction  efficacy  for  clinical  science  and  problem  solving 
achievement  imply  biomedical  science  knowledge  constructed  in  a  PBL  environment  is 
organized  for  better  deployment. 

Current  findings  suggest  the  success  of  PBL  is  not  limited  to  a  particular 
approach.  The  advocates  of  PBL  make  some  substantive  claims,  using  ill-structured 
problems,  using  re-iterative  methods,  and  so  on,  yet  none  of  these  claims  were  found 
uniquely  to  predict  the  instructional  effect  on  cognitive  outcomes.  Instances  of 
multicolinearity,  however,  suggest  some  substantive  features  may  converge  efficaciously 
as  a  function  of  context  and  desired  cognitive  outcome.  Despite  diverse  methods  of  PBL 
implementation,  problem  posing  and  integration  of  biomedical  and  clinical  sciences  in  a 
tutorial-group  setting  are  common  threads.  Since  direct  instruction  also  can  integrate 
biomedical  and  clinical  sciences,  further  research  is  required  to  examine  the  unique 
contribution  of  small-group  learning.  Since  the  ultimate  aim  of  medical  education  is  to 
graduate  competent  clinicians  who  are  expert  learners  and  problem  solvers,  examining 
what  distinguishes  the  structure  and  function  of  medical  knowledge  acquired  when 
undergoing  PBL  versus  LBL  may  be  more  insightful  for  informing  theory  and  practice 
than  global  contrasts  of  cognitive  and  non-cognitive  outcomes. 


CHAPTER  1 
INTRODUCTION 

Problem-based  learning  was  introduced  as  a  pedagogical  alternative  to  traditional 
lecture-based  medical  education.  Advocates  claim  problem-based  learning 
operationalizes  principles  of  modem  learning  theory  and  better  prepares  medical  students 
and  physicians  for  clinical  practice.  However,  research  comparing  the  relative  efficacy  of 
lecture-based  versus  problem-based  learning  has  produced  inconclusive  findings.  Present 
study  used  meta-analytic  methods  to  synthesize  results  of  primary  research  comparing 
cognitive  and  non-cognitive  outcomes  in  medical  students  and  physicians  who  undergo 
lecture-based  versus  problem-based  teaching  and  learning.  Principal  aim  of  current 
research  was  to  examine  the  main  effects  of  instructional  approach  on  five  broad 
constructs  including  biomedical  science  achievement,  clinical  science  achievement, 
problem-solving  achievement,  self-directed  learning  skills,  and  attitude  toward  learning. 
A  secondary  purpose  was  to  identify  those  outcome  measures,  methodological  and 
substantive  features  that  might  be  responsible  for  significant  variance  in  instructional 
effect.  Ultimately,  the  strengths  and  weaknesses  of  each  instructional  method  were 
examined  to  inform  practice  and  theory  and  serve  as  a  point  of  departure  for  future 
research. 

Background  and  Context 

Abraham  Flexner  (1910,  1925),  the  architect  of  modem  discipline-bound,  lecture- 
based  medical  education,  insisted  that  understanding  nomial  structure  and  function  were 


essential  to  learning  about  abnormal  structure  and  function  (1910).  "The  perception  of 

abnormality  or  irregularity  can  only  take  place  against  the  background  of  nomiality  or 

regularity"  (p.  106,  1925).  Moreover,  Flexner  rigorously  opposed  any  clinical  training 

during  the  first  two  years  of  medical  school. 

The  argument  in  behalf  of  mingling  clinical  and  pre-clinical  subjects  in  order  to 
interest  the  student  is  assuredly  not  convincing.  ...  A  clinical  illustration  may 
indeed  clarify  the  student's  comprehension  of  a  fundamental  principle  or  fact;  but, 
at  least  in  excess,  it  is  more  likely  to  excite  a  superficial  interest  in  symptom  or 
remedy,  (p.  1 12) 

Consequently,  a  hallmark  of  traditional  medical  school  teaching  and  learning  is 
the  separation  between  biomedical  science  and  clinical  science  education.  Preclinical 
education  is  dominated  by  discipline-oriented,  lecture-based  instruction  with  little 
intentional  instruction  in  the  application  of  biomedical  science  to  clinical  problems  (Bok, 
1984;  Ebert,  1992;  Schon,  1987;  Tosteson,  1994;  Swanson  &  Case,  1997).  Faculty 
members  in  the  biomedical  sciences,  although  primarily  non-physicians,  not  only  specify 
what  content  and  skills  students  must  master,  they  also  identify  the  resources  over  which 
students  are  to  achieve  mastery  (Distlehorst  &  Robbs,  1998).  Mastery  of  content  is 
assessed  primarily  using  examinations  incorporating  multiple-choice  questions  (Albanese 
&  Mitchell,  1993;  Distlehorst  &  Robbs;  Vernon  &  Blake,  1993),  and  the  principal 
learning  strategy  employed  by  successful  students  is  rote  memorization  (Lindblom- 
Ylanne  &  Lonka,  1999;  Regan-Smith  et  al.,  1994). 

During  clinical  science  training,  an  apprenticeship  model  guides  clinical 
education  as  students  undergo  clerkships  in  core  medical  specialties  including  internal 
medicine,  surgery,  obstetrics  and  gynecology,  pediatrics,  and  psychiatry.  Elective 
clerkships  are  available  to  senior  medical  students.   Physicians  provide  teaching  during 


clerkships  and  students  are  required  to  learn  in  less  structured  environments  approacliing 
those  of  clinical  practice. 

The  belief  that  traditional  pre-clinical  education  over-emphasized  rote 
memorization  and  failed  to  prepare  medical  students  adequately  for  clinical  experience 
prompted  the  introduction  of  problem-based  teaching  and  learning  (Barrows  &  Tamblyn, 
1980;  Neufeld  &  BaiTows,  1 974).  This  belief  was  echoed  when  a  panel  on  General 
Professional  Education  of  the  Physician  published  a  report  entitled  Physicians  for  the 
Twenty-first  Century  (1984)  which  concluded  that  "students  are  led  to  think  that  their 
education  depends  upon  memorizing  as  much  information  as  possible"  (Panel,  p.  1).  This 
deduction  was  substantiated  by  a  survey  of  453  senior  students  undergoing  traditional 
medical  education  at  five  universities  including  the  Universities  of  Florida  and  New 
Mexico,  Dartmouth,  Wake  Forest  University  (Bowman  Gray)  and  Rush  University 
(Regan-Smith  et  al.,  1994).  Forty-nine  percent  of  the  students  believed  they  learned  half 
or  more  of  the  biomedical  sciences  content  by  memorizing  without  understanding. 

More  importantly,  research  suggests  that  much  memorized  biomedical  science 
content  is  irretrievable,  or  determined  to  be  irrelevant  (Miller,  1961;  Neame,  1984;  Rico, 
Galindo,  &  Marset,  1981).  Miller  observed  poor  retention  of  biomedical  science  content 
in  medical  students  who  undei-went  lecture-based  learning  and  warned  that  information 
presented  via  teacher-centered  methods  would  "decrease  from  memory  at  the  same  rate 
as  nonsense  syllables"  (p.  153).  Rico  and  colleagues  observed  that  students  forgot  25% 
of  the  most  important  concepts  one  year  after  complefing  lecture-based  medical 
chemistry;  40%  after  three  to  five  years,  and  50%  after  eight  years.  In  addition, 
memorizing  information  with  little  or  no  practical  value  other  than  passing  examinations 


can  be  boring  and  possibly  even  demoralizing  for  students  (Des  Marchais  &  Vu,  1996; 
Schmidt,  1983;  Schon,  1987).  Lindblom-Ylanne  and  Lonka  (1999)  observed  that  the 
learning  environment  of  a  traditional  medical  school  can  give  misleading  cues  about  how 
to  study  efficiently  and  obliged  many  students  to  study  in  inadequate  ways,  driving  them 
towards  externally  regulated  and  superficial  learning.  Consequently,  many  students  were 
rendered  incapable  of  meaningful  learning  or  creative  thinking. 

Furthermore,  proponents  of  problem-based  learning  suggest  converting  from  a 
highly  structured  learning  environment  characterized  by  rote  memorization  (i.e.,  pre- 
clinical education)  to  a  more  self-directed  and  independent  learning  situation  (i.e.,  clinical 
clerkships)  may  be  problematic  for  students  (Schon,  1 983).  Indeed,  Patel,  Evans  and 
Groen  (1989)  describe  biomedical  and  clinical  science  knowledge  as  "two  worlds  apart" 
(p.  55).  Although  not  mentioned  in  their  report,  the  two-worlds  metaphor  employed  by 
Patel  et  al.  echoes  knowledge  distinctions  expressed  by  Karl  Popper  (1972)  and  is 
discussed  later. 

Magnani  (1997)  posits  that  medical  students  educated  in  a  traditional  manner  will 
encounter  difficulty  when  they  convert  from  biomedical  science  (unsituated  content, 
concerned  with  the  attributes  of  entities  such  as  organs,  pharmaceuticals  and  microbes)  to 
clinical  science  (situated  material,  concerned  with  attributes  and  health  problems  of 
people).  This  view  aligns  with  perspectives  expressed  by  some  situational  learning 
theorists  (Brown,  Collins,  &  Duguid,  1989;Greeno,  1998;  Lave  &  Wenger,  1991)  and 
will  be  discussed  later.  In  addition,  problem-based  learning  advocates  argue  students 
undergoing  lecture-based  learning  are  inadequately  prepared  when  encountering 
problems  requiring  the  transfer  of  their  learning  to  new  domains,  a  skill  required  to 


function  effectively  in  clinical  practice  (Barrows.  1986,  1990,  1996,  2000;  Boud  & 

Feletti,  1997).  This  is  a  consequence  Dewey  anticipated  (1933/1998)  when  he  suggested 

a  distinction  between  educating  memories  versus  educating  minds: 

It  is  not  enough  to  load  the  memory  with  statement  of  facts  and  laws  and  then 
hope  that  later  in  life  by  some  magic  the  mind  will  find  a  use  for  them.   Even 
general  principles,  when  merely  memorized,  stand  on  the  same  level  as  bare 
particular  facts.  Since  they  are  not  used  either  in  understanding  actual  objects  or 
events  or  in  giving  rise,  through  what  they  imply,  to  other  conceptual  meanings, 
they  are,  to  the  mind  that  memorizes  them  (falsely  called  learning),  mere  arbitrary 
items  of  information,  (pp.  185-185) 

McKegney  (1989)  rhetorically  concurred  with  these  investigators  when  he  wrote 
"The  experience  of  studying  in  medical  school  has  been  considered  to  be  analogous  to 
belonging  to  an  abusive  and  dysfunctional  family"  (p.  452).  And  Barrows  (2000)  posits, 
"This  is  not  only  educational  malpractice,  it  is  tragically  inefficient  when  you  consider 
how  much  energy  faculty  put  into  teaching  and  students  put  into  studying  during  these 
preclinical  years  to  result  in  such  a  small  yield"  (p.  .xiii).  This  cognitive  predicament  was 
anticipated  by  Whitehead  (1929)  who  argued  that  "inert  knowledge"  (p.  I)  is  not  just 
useless;  "it  is  harmful  -  Cornipiio  opliini.  pessima"  (p.  2)  (that  is,  corruption  of  the  best 
is  worst  of  all). 

Notwithstanding  philosophical,  anecdotal  and  evidence-based  admonishments  by 
opponents  of  traditional  medical  education,  71%  (84/1 18)  of  medical  schools  recognized 
by  the  Association  of  American  Medical  Colleges  still  employ  lecture-based,  discipline- 
oriented  preclinical  curricula  (Ripkey,  Swanson,  &  Case,  1998).  By  implication,  lecture- 
based  teaching  and  learning  are  considered  by  the  majority  of  medical  schools  to  be  the 
appropriate  pedagogical  method  to  facilitate  physician  training. 


Arguments  in  Favor  of  Traditional  Lecture-Based  Learning 

Reflecting  Flexner's  (1910,  1925)  model,  the  initial  two-years  of  traditional 
medical  school  currictila  in  North  America  incorporates  a  knowledge  reproduction  model 
in  which  disciplines  in  the  biomedical  sciences  are  the  primary  units  for  the  organization 
and  delivery  of  content.  This  content  is  presented  using  lecture-based  instruction,  or 
colloquially  speaking,  students  •■leam[ing]  by  being  told"  (Custers  &  Boshuizen,  2002,  p. 
1 70).  In  effect,  the  biomedical  science  disciplines  serve  as  a  context  for  structuring 
learning.  From  this  perspective  medical  knowledge  is  viewed  as  a  body  of  established 
facts  and  techniques  that  are  organized  hierarchically  and  context  free,  thus  able  to  be 
partitioned  and  transmitted  by  experts  to  novices  (Flexner;  Hemker,  1998;  Papa  & 
Harasym,  1999;  Schon,  1983,  1987),  Learning  is  viewed  as  the  successive  accumulation 
of  these  facts  and  techniques  pnmarily  by  listening,  observing,  memorizing  and 
practicing  (Schon). 

Widespread  endorsement  of  this  epistemological  stance  is  evidenced  by  the 
magnitude  of  preclinical  education  delivered  employing  a  knowledge  transmission 
paradigm  (Schon,  1983)  that  is  predominantly  lecture-based  (Ripkey  et  al.,  1998). 
Traditional  lecture  model  assumes  superior  knowledge  of  instructor  and  consequently 
situates  in  their  hands  responsibility  for  selecting  content  to  be  presented,  depth  of 
content  to  be  covered,  degree  of  interaction  between  content  and  practical  instances,  and 
duration  of  specific  content  coverage  (Dubin  &  Taveggia,  1968).  Traditional  biomedical 
science  lectures  are  organized  in  discrete  disciplines  and  sustaining  a  Flexnerian 
philosophy,  are  designed  not  to  complicate  theory  with  practice.  This  instructional 


approach  has  the  advantage  of  preserving  the  fundamentals  of  each  disciphne. 

Instruction  tends  to  be  well  structured  and  linear,  leading  to  expected  student  behavior 

and  response  (Papa  &  Harasyni,  1999).  Lectures  can  help  simplify  information, 

providing  an  entry  point  for  complex  content  (Rosenshine  &  Stevens,  1986).  That  is, 

lectures  facilitate  the  gradual  and  logical  development  of  complex  or  difficult  concepts 

and  theories  (Hemker,  1998).  Hemker  argues: 

Having  the  subject  explained  by  an  experienced  scientist  or  clinician  is 
particularly  useful  in  acquiring  the  attitude  and  skills  necessary  to  organize 
knowledge,  and  learning  to  distinguish  between  essential  and  accessory 
knowledge,  (p.  75) 

In  this  way,  the  aim  is  to  have  students  connect  the  new  material  to  previous  material  and 

to  practice  until  they  master  the  content.  Once  content  mastery  occurs,  the  students  can 

proceed  to  experiential  learning  activities  incorporating  inquiry  and  complex  problem 

solving.  Explicit  instruction  before  experiential  activities  reduces  chance  for 

development  of  misconceptions  during  initial  learning  (Rosenshine  &  Stevens). 

Moreover,  instructors  can  model  professional  practice  by  using  disciplinary 

exemplars  and  they  can  convey  the  intrinsic  value  of  the  content  through  their  enthusiasm 

(Rosenshine  &  Stevens,  1986).  According  to  Flexner  (1925),  a  good  lecturer  is  a 

"textbook  plus  a  personality"  (p.  181).  Also  supporting  this  stance,  Hirsch  (1997) 

believes  that  challenging  subject  content  "can  be  taught  in  a  lively,  demanding  way"  (p. 

43)  through  lectures.  Lectures  present  minimal  risk  to  students  and  appeals  to  those  who 

prefer  to  learn  by  listening  or  those  whose  own  learning  strategies  are  ineffective 

(Hirsch).  Accordingly,  students  develop  a  comfort  level  and  need  only  to  depend  on  a 

syllabus  and  instructor  for  guidance  (Barrows,  1996,  2000).  Furthermore,  instructors 


g 

themselves  feel  more  comfortable  and  perform  better  in  a  lecture-based  system,  and  it  is 
cost  effective  and  less  demanding  as  well  (Vemon  &  Blake,  1993). 
Arguments  in  Favor  of  Problem-Based  Learning 

Interestingly,  seven  years  before  Flexner's  report  to  the  Carnegie  Foundation, 

William  Osier  (1903)  argued  for  a  more  practice-based  medical  education  when  he  wrote 

In  what  may  be  called  the  natural  method  of  teaching  the  student  begins  with  the 
patient,  continues  with  the  patient,  and  ends  with  his  studies  with  the  patient, 
using  books,  and  lectures  as  tools,  as  a  means  to  an  end.  .  .  .  take  him  from  the 
lecture-room,  take  him  from  the  amphitheater,  -  put  him  in  the  out-patient 
department  -  put  him  in  the  wards.  ...  the  best  teaching  is  that  taught  by  the 
patient  himself  (p.  49-50) 

Apparently,  for  Osier,  no  inedical  teaching  and  learning  should  be  done  without  a  patient 

context.  Reflecting  epistemology  advanced  by  Osier,  proponents  of  problem-based 

teaching  and  learning  claim  that  to  understand  the  principles  and  practices  of  medicine, 

knowledge  construction  grounded  in  the  integration  of  biomedical  science  and  clinical 

science  embedded  in  the  problems  reflecting  clinical  practice  is  prerequisite  (Bartows, 

1986,  1990,  1996,  2000;  Barrows  &  Tamblyn,  1980;  Boud  &  Feletti,  1997;  Neufeld  & 

Barrows,  1974).  Consequently,  problem-based  learning  experiences  are  designed  to 

integrate  biomedical  and  clinical  sciences  into  the  context  of  authentic  clinical  practice. 

Learning  experiences  are  intended  to  give  students  an  opportunity  as  well  as  a 

responsibility  to  establish  what  they  need  to  know  and  master  the  process  of  inquiry 

necessary  for  resolving  health  care  problems.  These  learning  experiences  are  tailored 

toward  achieving  educational  expectations  and  objectives  (Barrows;  Schon,  1983,  1987). 

By  design,  problem-based  instruction  is  student-centered  with  tutorial  groups  that 

emphasize  collaborative  learning  (Barrows;  Boud  &  Feletti;  Stage,  Muller,  Kinzie,  & 

Simmons,  1998).  Although  an  instructor  (tutorj  does  have  considerable  responsibility  in 


facilitating  inquiry,  students  are  expected  to  gradually  assume  responsibility  for  their  own 
learning.  With  appropriate  experience  and  guided  practice,  it  is  postulated  that  students 
eventually  will  gain  full  independence  with  the  instructor  gradually  becoming  a  co- 
worker. The  emphasis  is  on  active  student  acquisition  of  infomiation  and  skills  suitable 
to  their  ability,  level  of  experience,  and  educational  needs.  Ideally,  students  determine 
the  best  methods  for  learning,  find  the  necessary  resources,  set  the  pace,  and  structure  the 
acquisition  of  information  within  an  activity  designed  to  resolve  a  health  care  problem 
(Barrows).  In  this  manner,  problem-based  learning  focuses  on  diversity  rather  than 
uniformity.  It  fosters  the  acquisition  of  a  rich,  integrated  and  organized  medical 
knowledge  base  that  is  linked  to  authentic  health  care  problems  and  encourages 
individual  and  collaborative  processes  of  reasoning  (inquiry).  This  pedagogical  approach 
appears  to  be  suitable  for  preparing  students  for  lifelong  learning  (Barrows;  Savery  & 
Duffy,  1995;  Zimmerman  &  Lebeau,  2000).  Moreover,  problem-based  teaching  and 
learning  operationalizes  adult  learning  theory  (Knowles,  1984),  the  principles  of 
knowledge  activation  and  encoding  specificity  (Anderson,  1993,  1996;  Ausubel  I960; 
Chi,  DeLeeuw,  Chiu,  &  LaVancher,  1994;  Newell  &  Simon,  1972),  deep-processing 
(Bloom,  1984;  Craik  &  Lockhart,  1972;  Newblc  &  Clark,  1986),  social  constructivism 
(Cobb,  2002;  Prawat,  1993;  Savery  &  Duffy;  Vygotsky,  I926/I997),  and  situated 
cognition  (Brown  et  al.,  1989;  Greeno,  1998;  Lave  &  Wenger,  1991;  Schmidt,  1983, 
1993;  Schmidt  et  al.,  1996;  Schmidt,  Nomian,  &  Boshuizen,  1990). 
Synthesis  of  Arguments 

Using  both  problem-based  and  lecture-based  teaching  and  learning  methods, 
instructors  may  prepare  what  theyjudge  to  be  appropriate  learning  objectives,  learning 
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resources,  choice  of  pathways,  and  evaluation  materials  that  reflect  their  particular 

experience  and  knowledge.  Traditional  lecture-based  instruction  begins  by  exposing 

students  to  disciplinary  knowledge,  and  then  assigning  problems  to  promote  the  use  and 

mastery  of  that  knowledge.  Traditional  lecture-based  instruction  typically  employs  only 

one  facet  of  direct  instruction  (Rosenshine  &  Stevens,  1986),  that  is,  didactics  begin  by 

transmitting  disciplinary  knowledge  to  students,  and  then  assigning  problems  to  promote 

the  use  and  mastery  of  that  knowledge.  Each  biomedical  science  discipline  (e.g., 

anatomy,  biochemistry,  phamiacology,  and  physiology)  has  a  separate  block  of  curricular 

time.  Dewey  (1933/1998)  described  such  isolation  of  facts  from  meaning  when  he  wrote. 

Their  minds  are  loaded  with  disconnected  piecemeal  items  that  are  accepted  on 
hearsay  or  authority.  .  .  .  and  no  attempt  is  made  to  interpret  it  in  relation  to  what 
it  does,  how  it  was  caused,  and  what  it  stands  for.  (p.  184) 

For  Dewey,  the  rationale  for  this  epistemic  stance  is  that 

It  assumes  that  logical  quality  belongs  only  to  organized  knowledge  and  that  the 
operations  of  the  mind  become  logical  only  through  absorption  of  logically 
fonnulated,  ready-made  material.  In  this  case,  the  logical  fonnulations  are  not  the 
outcome  of  any  process  of  thinking  that  is  personally  undertaken  and  carried  out; 
the  formation  has  been  made  by  another  mind  and  is  presented  in  finished  form, 
apart  from  the  process  by  which  it  was  arrived  at.  Then  it  is  assumed  that  by 
some  magic  its  logical  character  will  be  transferred  into  the  minds  of  pupils,  (p. 
80) 

Students  in  such  a  learning  environment,  were  characterized  as  "spectators"  by  Dewey 

(1916,  p.  146).  Moreover,  mastery  learning  is  based  on  the  assumption  that  once  medical 

students  exhibit  competency,  they  will  remain  competent.  However,  retaining  knowledge 

acquired  during  medical  school  for  a  lifetime  is  improbable  (Bok,  1984),  if  not 

undesirable  given  that  the  half-life  of  medical  education  now  is  estimated  to  be  five-years 

(Gorman,  Meier,  Rawn,  &  Krummel,  2000).  Clearly,  the  knowledge  reproduction  model 

is  inadequate.  Thus,  information  management,  rather  than  infomiation  mastery,  appears 
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to  be  a  more  appropriate  aim  of  preclinical  medical  education.  Furthermore,  knowledge 
reproduction  models  devalue  content  learned  and  foster  a  situation  in  which  learners  are 
expected  to  encode  knowledge  implicitly  grounded  in  theory  when  the  deployment  and 
value  of  that  knowledge  are  explicitly  attributed  to  practice  (Barrows,  2000;  Cobb,  2002; 
Perkins  &  Salomon,  1989;  Savery  &  Duffy,  1995;  Schon,  1983,  1987). 

Because  clinical  science  problems  rarely  involve  isolated  disciplinary  knowledge, 
ex  post  facto  the  medical  student  has  to  synthesize,  integrate,  and  eventually  apply  to 
practice  these  isolated  blocks  of  knowledge.  Thus,  knowledge  and  skills  acquisition 
occurs  in  an  environment  dominated  by  instruction,  not  construction,  in  a  process  I 
characterize  as  "retrospective  applicability."  Consequently,  for  traditional  medical 
students,  two  types  of  knowledge  transfer  must  occur  (Patel,  Groen,  &  Scott,  1988;  Patel 
et  al.,  1989;  Patel,  Groen,  &  Norman,  1991;  Schmidt,  DeGrave,  de  Voider,  Moust  & 
Patel,  1989).  Initially,  biomedical  science  knowledge  must  be  transferred  to  clinical 
science  knowledge  (i.e.,  the  integration  of  biomedical  and  clinical  science  knowledge). 
Then,  this  integrated  biomedical  and  clinical  sciences  knowledge  must  be  transferred  to 
representing  and  solving  the  health  care  problems  of  individual  patients. 

Rather  than  assume  a  mythical  transfer  (Dewey,  1 933/1 998)  as  students  convert 
fi'om  a  biomedical  science  world  to  a  clinical  science  world  (Patel  et  al.,  1989;  Popper, 
1972;  Schon,  1983),  problem-based  teaching  and  learning  assume  that  there  is  no 
distinction  between  theory  and  practice.  Accordingly,  there  is  a  reciprocal  relationship 
between  biomedical  and  clinical  sciences  with  one  informing  the  other,  all  framed  within 
the  context  of  health  care  problems  (Barrows,  1983,  1986,  1990,  1996,  2000;  Barrows  & 
Tamblyn,  1980;  Boud  &  Feletti,  1997;  Neufeld  &  Barrows,  1974). 
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Health  care  problems  stimulate  a  process  of  inquiry  and  knowledge  construction 
in  which  biomedical  knowledge  feeds  into  clinical  knowledge  (Barrows,  2000;  Cobb, 
2002;  van  de  Wiel,  Boshuizen,  &  Schmidt,  2000).  Problems  are  designed  to  provide 
students  with  knowledge  that  has  clinical  relevance  and  therapeutic  significance  (the 
purpose  of  knowledge)  while  encouraging  them  to  assume  responsibility  for  their 
learning  (personally  constructed  meaning),  that  is,  knowledge  as  design  (Perkins,  1986). 
Viewed  from  this  framework,  knowledge  as  infomiation  is  passive,  while  knowledge  as 
design  becomes  active.  Perkins  suggests  "Knowledge  disconnected  from  context  is 
nothing  short  of  truth  mongering  .  .  ."  (p.  1).  Purpose  and  meaning  making  should  act  as 
psychological  tools  (Vygotsky,  1926/1997)  for  understanding  concepts.  For  Perkins,  the 
aim  of  professional  education  is  to  teach  knowledge  as  a  design  for  sense  making,  a 
learning  process  I  characterize  as  "concurrent  applicability."  Accordingly,  in  acquiring 
knowledge  learners  need  to  invoke  a  cognitive  auditing  method  to  ascertain:  (a)  Is  this 
fact  impoilant?,  (b)  Is  it  relevant?,  (c)  Does  it  have  personal  meaning?,  and  (d)  Is 
knowledge  linked  to  a  purpose?  (Perkins,  p.  4).  From  this  perspective,  "purpose"  is  the 
principal  tool  to  reveal  evidential  value  for  triaging  and  understanding  infomiation.  In 
other  words,  information  calibrated  with  praginatism  guides  meaningful  knowledge 
construction.  Indeed,  "when  a  piece  of  infomiation  is  connected  to  a  purpose,  it  becomes 
design-like"  (Perkins,  p.  4).  Extending  Deweyan  (1933/1998)  philosophy,  problem- 
based  teaching  and  learning  is  concemed  with  educating  the  "mind"  versus  educating  the 
"memory."  Indeed  for  Dewey,  "learning  is  leaming  to  think"  (p.  78). 
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Legitimacy  of  Problem-Based  Learning  Is  Hampered  by  Equivocal  Evidence 

Some  educators  argue  that  the  focus  of  problem-based  instruction  on  cognitive 
processes  and  student-directed  learning  results  in  the  acquisition  of  biomedical  science 
knowledge  they  label  as  "basic  science  lite"  (Swanson  &  Case,  1997,  p.  73).  Morgan 
(1995)  notes  that  when  graduates  of  the  problem-based  learning  program  at  McMaster 
University  go  to  other  medical  schools  for  their  clinical  training,  they  have  less 
biomedical  science  knowledge  initially  than  other  Canadian  graduates,  although  "they 
soon  catch  up"  (p.  1643).  Morgan  suggests  ".  .  .  the  General  Medical  Council  [the 
governing  board  for  medical  education  in  the  United  Kingdom],  with  its  enthusiasm  for 
change,  would  do  well  to  remeinber  Burke's  dictum,  'To  innovate  is  not  to  reform'"  (p. 
1643). 

Research  comparing  knowledge  acquisition,  problem-solving  skills,  and  self- 
directed  learning  in  medical  students  and  physicians  who  have  undergone  problem-based 
versus  lecture-based  instruction  is  contradictory.  Three  literature  reviews  (Albanese  & 
Mitchell,  1993;  Berkson,  1993;  Colliver,  1999)  and  one  meta-analysis  (Vernon  &  Blake, 
1993)  comparing  the  curricular  effects  of  problem-based  learning  (PBL)  versus  lecture- 
based  learning  (LBL)  have  been  published.  Three  of  these  four  reports  emphasize 
different  instructional  characteristics  and  include  non-physician  training  programs 
(Albanese  &  Mitchell;  Berkson;  Venion  &  Blake).  Albanese  and  Mitchell  classify  and 
list  quantitative  results  (unstandardized  and  unweighted  effect  sizes)  of  research 
conducted  from  1972  to  1992  revealing  that  PBL  students  had  lower  biomedical  science 
achievement.  They  conclude  that  PBL  students  exhibit  an  "incomplete  understanding  of 
the  basic  sciences"  (p.  77),  but  PBL  students  found  learning  more  "nurturing  and 
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enjoyable"  (p.  78).  The  authors  suggest  that  "caution  be  exercised  in  making 
comprehensive,  curriculum-wide  conversions  to  PBL"  (p.  52)  until  further  research 
establishes  guidelines  regarding  the  degree  of  faculty  direction,  the  necessary  cost 
effectiveness  of  implementing  PBL,  and  establishes  reasons  for  biomedical  sciences 
content  weakness  in  PBL  students. 

Vernon  and  Blake  (1993),  who  employed  meta-analytic  methods  to  synthesize  the 
results  of  coiTiparative  instructional  research  from  1970  to  1992,  reported  similar 
findings.  Quantified  with  effect  size  (ES),  they  found  PBL  medical  students  had  inferior 
biomedical  science  achievement  (ES  =  -0.18),  slightly  better  clinical  science  achievement 
(ES  =  0.08)  as  ineasured  by  national  licensing  examinations,  better  supervisor  rated 
clinical  performance  (ES  =  0.28),  and  expressed  greater  self-rated  satisfaction  with  their 
learning  environment  (ES  =  0.55). 

Berkson  (1993)  employed  a  traditional,  nanative  method  to  review  literature  from 
1983  to  1991  and  concluded  "the  graduate  of  PBL  is  not  distinguishable  from  his  or  her 
traditional  counterpart.  In  addition,  the  experience  of  PBL  can  be  stressful  for  students 
and  faculty.  And  implementation  of  PBL  may  be  unrealistically  costly"  (p.  S85). 
Berkson  concluded  PBL  curricula  were  unlikely  to  surpass  traditional  curricula  in 
imparting  problem-solving  expertise,  acquisition  of  knowledge,  enhancing  motivation, 
improving  self-directed  learning,  or  providing  experiences  that  are  more  enjoyable. 

Colliver  (1999),  who  conducted  a  narrative  review  of  studies  involving  only 
medical  schools  with  curriculum-wide  implementation  of  PBL  from  1992  to  1998,  found 
eight  studies  and  concluded  there  "is  no  convincing  evidence  that  PBL  iinproves 
knowledge  base  or  clinical  performance"  (p.  259). 
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Others  present  evidence  indicating  that  PBL  compared  to  LBL  students  acquire 
less  content  knowledge  and  exhibit  a  tendency  to  develop  misconceptions  and  errors 
while  applying  biomedical  science  knowledge  to  clinical  problems  (Bernstein,  Tipping, 
Bercovitz,  &  Skinner,  1995).  Moreover,  Ripkey  and  associates  (1998)  reviewed 
interschool  differences  in  United  States  Medical  Licensing  Examination  (USMLE)  Step  1 
scores  and  concluded  that  curriculum  type  contributed  little  (approximately  6%  of  the 
variance)  to  observed  variations  in  medical  students'  acquisition  of  biomedical  science 
knowledge.  Unfortunately,  the  curricular  effects  of  schools  incorporating  nontraditional 
programs  (including  problem-based  instruction)  could  not  be  distinguished  because  the 
sample  size  was  too  small. 

Since  student  performance  on  licensing  examinations  is  considered  a  benchmark 
of  curriculum,  the  degree  to  which  knowledge  and  problem-solving  skills  are  acquired  in 
a  PBL  curriculum  has  manifested  considerable  concern  and  debate  (Albanese,  2000; 
Colliver,  2000;  Norman  &  Schmidt,  2000).  Students  in  the  United  States  take  two  of 
these  examinations  (USMLE  Step  1  and  Step  2)  during  medical  school  and,  depending  on 
their  school,  may  be  required  to  pass  Step  1  to  advance  to  year  three  of  medical  school 
and  Step  2  to  graduate.  In  addition,  the  results  of  USMLE  Step  1  and  Step  2 
examinations  significantly  affect  selection  for  graduate  medical  education  (e.g., 
residency).  Step  3  must  be  passed  before  license  to  practice  is  granted.  Given  the  high 
stakes  of  these  examinations,  efforts  to  improve  curriculum  often  engender  concern  that 
learning  instead  might  inadvertently  be  impaired. 

In  summary,  only  one  meta-analytic  investigation  has  compared  cognitive  and 
noncognitive  learning  outcomes  involving  health  care  professionals  (Vernon  &  Blake, 
1993).  Results  of  that  meta-analysis  regarding  cognitive  outcomes  were  mixed  and 
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incorporated  data  from  licensing  examinations  that  have  changed  significantly 
(O'Donnell,  Obenshain,  &  Erdmann,  1993;  Williams,  1993)  over  the  years.  In  1992, 
USMLE  replaced  the  National  Board  of  Medical  Examiners  (NBME)  and  shifted  the 
emphasis  from  measuring  factual  and  abstract  biomedical  scientific  knowledge  to 
assessing  the  application  of  biomedical  knowledge  (O'Donnell  ct  al.).  Moreover, 
previous  meta-analysis  did  not  statistically  distil  or  analyze  effect  size  variance  beyond 
its  main  effects.  Therefore,  there  has  been  no  systematic  accounting  of  variance 
associated  with  outcome  measures,  methodologic  features,  or  substantive  features  of 
problem-based  teaching  and  learning  in  medical  education. 
Statement  of  Problem 
Despite  cognitive  and  instructional  theory  predicting  the  unique  superiority  of 
problem-based  teaching  and  learning,  the  research  literatui'c  is  conflicting  with  regard  to 
the  relative  efficacy  of  problem-based  versus  lecture-based  instruction.  In  addition, 
efficacy  is  exacerbated  by  the  variety  of  methods  used  to  implement  problem-based 
instruction  (Barrows,  2000;  Blumberg,  Michael,  &  Zeitz,  1990).  Consequently,  some 
medical  educators  have  recommended  caution  when  considering  conversion  to  problem- 
based  leai-ning  (Albanese  &  Mitchell,  1993;  Berkson,  1993;  Colliver,  2000;  D'Eon,  1999; 
Morgan,  1995).  Risks  cited  include  inferior  biomedical  science  achievement  exhibited 
by  PBL  versus  LBL  students  as  evidenced  by  results  of  licensing  examinations.  In 
addition,  some  evidence  suggests  that  compared  to  their  traditional  counterparts,  PBL 
students  exhibit  greater  academic  stress  and  perceive  that  they  acquire  less  content 
knowledge.  Cited  benefits  include  better  self-directed  learning  skills  and  more  favorable 
attitudes  toward  learning,  although  these  observations  are  not  uniform  across  studies. 
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Consequently,  at  issue  is  wliether  student-centered,  problem-based  instruction  can 
facilitate  efficient  and  effective  biomedical  and  clinical  sciences  knowledge  acquisition, 
enhance  the  development  of  problem  solving  skills  as  well  as  self-directed  learning  skills, 
and  stimulate  a  positive  attitude  toward  learning  more  than  lecture-based  pedagogical 
methods.  Since  many  medical  schools  are  currently  contemplating  significant  ctirricular 
revision,  it  is  important  to  examine  the  efficacy  of  problem-based  instruction  (D'Eon, 
1999;  Harden,  Grant,  Buckley  &  Hart,  2000;  Kaufman,  2000;  Prideaux,  2000;  Rothman, 
2000). 

Medical  education  research  often  fails  to  render  clear,  generalizable  results,  due  in 

part  to  insufficient  sample  size  and  idiosyncratic  methodology  (Harden  et  al.,  2000).  In 

addition,  important  differential  effects  are  not  apparent  in  individual  studies  (Dubin  & 

Taveggia,  1968;  Lipsey  &  Wilson,  1993).  Research  literature  involving  problem-based 

learning  is  no  exception  (Aaron  et  al.,  1998;  Albanese  &  Mitchell,  1993;  Berkson,  1993; 

Colliver,  1999;  Vemon  &  Blake,  1993).  Indeed,  for  Aaron  and  associates: 

Despite  a  substantial  amount  of  research  in  a  variety  of  centers  of  medical 
education,  it  has  been  very  difficult  to  provide  objective  evidence  that  a  medical 
education  system  which  uses  PEL  generates  graduates  who  have  significant 
differences  in  knowledge,  skills,  or  behavior  from  those  emerging  from  a 
traditional  didactic  system,  (p.  86) 

Inconsistencies  in  primary  research  findings  motivate  this  study.  The  research 

aim  is  to  quantify  the  main  effects  of  problem-based  versus  lecture-based  instruction  with 

regard  to  biomedical  science  achievement,  clinical  science  achievement,  problem-solving 

skills,  self-directed  learning  skills,  and  positive  learning  attitudes.  A  secondary  research 

objective  is  to  examine  the  utility  of  theory  guiding  problem-based  teaching  and  learning 

by  analyzing  variations  in  effect  size  estimates  as  a  function  of  outcome  measures. 
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research  methods,  and  pedagogical  methods.  The  residts  should  assist  us  in  deciding  the 
efficacy  of  problem-based  instruction  and  if  theories  of  cognition  can  guide  teaching  and 
learning  to  improve  both  medical  student  achievement  and  the  cognitive  and 
noncognitive  processes  affecting  achievement. 

Research  Questions 

This  investigation  will  analytically  compare  cognitive  and  non-cognitive 
educational  outcomes  (dependent  variables  quantified  as  standardized  weighted  effect 
size  estimates)  in  medical  students  and  physicians  who  undergo  problem-based  versus 
lecture-based  teaching  and  learning.  The  investigation  will  examine  the  main  effects  of 
instructional  approach  on  five  broad  constructs  including  biomedical  science 
achievement,  clinical  science  achievement,  problem-solving  achievement,  self-directed 
learning  skills,  and  attitude  toward  learning.  Theoretically  grounded  conditional  effects 
will  be  addressed  to  identify  those  outcome  measures,  methodological  and  substantive 
features  that  may  be  responsible  for  significant  variance  heterogeneity.  Outcome  features 
are  concerned  with  the  nature  of  dependent  measures  analyzed;  methodological  features 
are  concerned  with  the  quality  of  research  design;  substantive  features  are  characteristics 
of  instructional  intervention,  the  fidelity  of  problem-based  instruction,  and  environment 
(e.g.,  scope  of  PBL,  PBL  taxonomy,  problem-structure,  tutor  characteristics,  origin  of 
learning  issues,  and  responsibility  for  self-directed  learning). 

The  research  is  guided  by  the  following  questions: 

1 .  Is  there  a  positive  effect  size  for  biomedical  science  achievement  in  students  and 
physicians  who  undergo  medical  education  with  problem-based  compared  to 
lecture-based  teaching  and  learning? 

2.  Is  variation  in  biomedical  science  achievement  effect  size  estimates  greater  than 
sampling  error  warranting  subgroup  analyses  to  test  for  potential  moderation  by 
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a  priori  defined  outcome  measures,  methodological  features,  and  substantive 
features  of  problem-based  teaching  and  learning? 

3.  Is  there  a  positive  effect  size  for  clinical  science  knowledge  achievement  in 
students  and  physicians  who  undergo  medical  education  with  problem-based 
compared  to  lecture-based  teaching  and  learning? 

4.  Is  variation  in  clinical  science  achievement  effect  size  estimates  greater  than 
sampling  error  warranting  subgroup  analyses  to  test  for  potential  moderation  by  a 
priori  defined  outcome  measures,  methodological  features,  and  substantive 
features  of  problem-based  teaching  and  learning? 

5.  Is  there  a  positive  effect  size  for  problem  solving  achievement  in  students  and 
physicians  who  undergo  medical  education  with  problem-based  compared  to 
lecture-based  teaching  and  learning? 

6.  Is  variation  in  problem-solving  achievement  effect  size  estimates  greater  than 
sampling  error  warranting  subgroup  analyses  to  test  for  potential  moderation  by  a 
priori  defined  outcome  measures,  methodological  features,  and  substantive 
features  of  problem-based  teaching  and  learning? 

7.  Is  there  a  positive  effect  size  for  self-directed  learning  in  students  and  physicians 
who  undergo  medical  education  with  problem-based  compared  to  lecture-based 
teaching  and  learning? 

8.  Is  variation  in  self-directed  learning  effect  size  estimates  greater  than  sampling 
error  warranting  subgroup  analyses  to  test  for  potential  moderation  by  a  priori 
defined  outcome  measures,  methodological  features,  and  substantive  features  of 
problem-based  teaching  and  learning? 

9.  Is  there  a  positive  effect  size  for  attitude  toward  learning  in  students  and 
physicians  who  undergo  medical  education  with  problem-based  compared  to 
lecture-based  teaching  and  learning? 

10        Is  variation  in  attitude  toward  learning  effect  size  estimates  greater  than  sampling 
error  warranting  subgroup  analyses  to  test  for  potential  moderation  by  a  priori 
defined  outcome  measures,  methodological  features,  and  substantive  features  of 
problem-based  teaching  and  learning? 

Definition  of  Terms 

Similar  instructional  methods  and  outcomes  are  operationalized  differently  in  the 
research  literature  synthesized  in  this  meta-analysis.  Recognizing  inherent  operational 
variability,  the  following  definitions  are  used  throughout  this  investigafion. 
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Lecture-based  learning  (LBL).  Is  defined  as  a  discipline-oriented,  teacher- 
centered,  expository  transmission-of-content  model.  Biomedical  science  disciplines  are 
the  primary  units  for  organization  and  content  is  dehvered  primarily  through  lectures, 
which  may  be  supplemented  with  other  fomis  of  direct  instruction  (e.g.,  demonstration 
laboratories,  faculty  guided  skills  practice,  expert  guided  seminars)  (Rosenshine  & 
Stevens,  1986).  Typically,  disciplines  include  anatomy,  behavioral  science, 
biochemistry,  immimology,  microbiology,  pathology,  pharmacology,  and  physiology. 
An  assumption  of  this  pedagogical  method  is  that  learning  results  by  the  accumulation  of 
infomiation  transmitted  by  the  instructor  (expert)  to  the  student  (novice).   LBL  is  a 
knowledge  reproduction  model  with  the  instructor's  expected  learning  outcomes 
expressed  in  behavioral  objectives.  Biomedical  science  instructors  not  only  specify  what 
content  and  skills  must  be  mastered,  they  also  identify  the  resources  over  which  students 
are  to  achieve  mastery.  Although  there  may  be  some  performance-based  assessment 
employing  human  and  computer  simulations  (e.g..  Objective  Structured  Clinical 
Examinations),  content  mastery  is  the  aim  and  is  assessed  primarily  by  examinations 
dominated  by  multiple-choice  questions.  Lecture-based  instniction  may  be  implemented 
as  part  of  a  course,  an  entire  course,  or  an  entire  curriculum.  Curriculum-wide 
implementation  involves  preclinical  content  (biomedical  sciences).  This  occurs  within 
the  first  two  years  of  medical  school  in  the  United  States  and  Canada  and  the  first  three  to 
four  years  in  Europe  and  Australia. 

Problem-based  learning  (PBL).  Is  defined  as  a  pedagogical  method  that 
assumes  learning  results  from  the  process  of  working  toward  the  understanding  or 
resolution  of  a  health  care  problem.  PBL  may  be  implemented  as  part  of  a  course,  an 
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entire  course,  or  an  entire  curriculum.  Altliough  preclinical  core  content  and  temporal 

constraints  are  similar  to  lecture-based  cunicula,  curriculum  wide  implementation  of 

PBL  aims  to  foster  biomedical  science  knowledge  acquisition  and  understanding  while 

students  negotiate  authentic  clinical  practice  problems. 

The  implementation  of  PBL  varies  considerably.  Consequently,  Barrows  (1986) 

proposed  a  taxonomy  framed  according  to  obtainable  objectives  of  problem-based 

learning  including  structuring  knowledge  in  clinical  contexts  (SCC),  nurturing  the 

clinical  reasoning  process  (CRP),  the  cultivation  of  efficacious  self-directed  learning 

skills  (SDL),  and  enhancing  motivation  for  learning  (MOT).  For  the  purpose  of  this 

research,  the  fidelity  of  problem-based  instruction  is  defined  according  to  the  following 

taxonomy  (adapted  from  Barrows,  1986,  pp.  483-484). 

L  Lecture-based  cases:  Students  are  presented  with  information  in  lectures  and  one 

or  two  patient  cases  are  presented  to  demonstrate  the  relevance  of  the  information 
provided.  No  inquiry  skills  are  required. 

2.  Case-based  lectures:  Students  are  given  patient  case  histories  as  advanced 
organizers  for  content  to  be  presented  in  a  subsequent  lecture.  Presumably, 
activation  of  prior  knowledge  is  accomplished,  but  no  self-directed  learning  is 
encouraged  formally. 

3.  Case  method:  Students  are  given  a  comprehensive  patient  case  to  be  studied  in 
preparation  for  class  discussion.  The  purpose  is  to  organize  and  synthesize 
information  to  direct  the  application  of  learning.  Class  discussion  amalgamates 
student-directed  and  teacher-directed  learning.  This  technique  incorporates 
elements  of  hypothesis  generation,  information  analyses,  and  clinical  reasoning. 
However,  the  patient  case  information  and  pattern  of  discussion  questions  (often 
Socratic  style)  are  synthesized  and  organized,  respectively,  by  the  instructor  for 
student  consumption. 

4.  Modified  case-based  method:  This  approach  differs  from  the  case  method  in  that 
patient  case  infonnation  is  less  structured  and  students  may  be  provided  with 
numerous  alternatives  to  investigate.  However,  partial  prospective  organization 
of  information  obviates  a  complete  and  unrestricted  inquiry  by  students. 
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5.  Problem-based  learning:  Students  are  presented  with  a  patient's  presenting 

symptoms.  Patient's  problems  are  more  ambigtious  than  those  encountered  with 
case  method  where  the  case  writer  exerts  more  influence  on  structuring  the  issues. 
Consequently,  students  are  encouraged  to  self-direct  their  inquiries  grounded  in 
their  prior  knowledge  unconstrained  by  the  curriculum,  instmctor,  or  peers. 

6.    Closed-loop  or  reiterative  problem-based  learning:  This  approach  extends  the 
problem-based  method  in  that  following  initial  generation  of  multiple  hypotheses 
of  what  might  be  happening  to  the  patient;  a  reflective  phase  complements  the 
problem-based  format.  That  is,  learning  issues  are  generated  collaboratively  from 
the  problem-challenge,  and  then  learning  issues  are  addressed  via  self-directed 
learning.  One  or  two  days  later,  students  meet  to  share  the  results  of  self-directed 
learning  as  they  re-visit  the  problem-challenge. 

These  problem-based  learning  methods  and  their  associated  comparative  power 

hypothesized  to  achieve  desired  educational  objectives  are  summarized  in  Table  1-1 

(adapted  from  Barrows,  p.  483). 

Table  1-1.  Problem-based  learning  methods  operationalized  as  a  function  of  relative 
power  to  achieve  educational  objectives 


PBL  Method 

Relative 

power  to  achieve 

educational 

obj 

ectives 

sec 

CRP 

SDL 

MOT 

Lecture-based  cases 

1 

1 

0 

1 

Case-based  lectures 

2 

2 

0 

2 

Case  method 

3 

3 

3 

4 

Modified  case-method 

4 

3 

3 

5 

Problem-based 

4 

4 

4 

5 

Reiterative  problem-based 

5 

5 

5 

5 

Note.  PBL  =  problem-based  learning;  SCC  =  structuring  knowledge  in  clinical 
contexts;  CRP  =  nurturing  clinical  reasoning  process;  SDL  =  cultivation  of  efficacious 
self-directed  learning  skills;  MOT  =  enhancing  motivation  for  learning.  Relative 
power  to  achieve  educational  objectives  is  indexed  by  Barrows  (1986)  from  0  to  5. 

Biomedical  science  acliievement.  Is  measured  as  performance  on  national 

medical  licensing  examinations  or  locally  developed  course  specific  examinations. 

Comprehensive  biomedical  science  knowledge  in  the  United  States  was  assessed  using 

National  Board  of  Medical  Examiners  (NBME),  Part  1  until  1992,  and  thereafter.  United 

States  Medical  Licensing  Examination  (USMLE),  Step  1.  Step  1  contains  over  700 

multiple-choice  questions  that  assess  student  knowledge  of  important  concepts  of  science 
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considered  basic  to  the  practice  of  medicine,  with  special  emphasis  on  biomedical  science 
knowledge,  principles  and  mechanisms  underlying  health,  disease,  and  modes  of  therapy. 
Presumably  Step  1,  usually  completed  late  in  the  second  year  of  medical  school,  ensures 
mastery  of  the  sciences  underlying  the  safe  and  competent  practice  of  medicine  in  the 
present,  as  well  as  the  scientific  principles  required  for  maintaining  competence  through 
lifelong  learning.  Locally  developed  assessments  of  biomedical  knowledge  include 
multiple  choice  questions,  short  answer,  and/or  essay.  When  hybrid  formats  are 
employed  for  assessment,  they  are  referred  to  as  multi-method,  multi-modal  or  mixed 
assessments. 

Clinical  science  achievement.  Is  measured  as  performance  on  national  medical 
licensing  examinations;  locally  or  nationally  developed  written  examinations;  objective 
structured  clinical  examinations  (OSCE);  multi-station,  standardized-patient-based 
clinical  performance  examinations  (CPX);  and  supervisor  ratings.  For  instance,  the 
clinical  science  knowledge  base  typically  is  assessed  systematically  using  the  subject 
examinations  of  the  NBME-II  (i.e.,  NBME-II  Shelf-board  examinations)  administered  to 
all  medical  students  in  the  United  States  during  clerkships  (e.g..  Family  Medicine, 
Internal  Medicine,  Obstetrics/Gynecology,  Pediatrics,  Psychiatry,  and  Surgery).  Clinical 
science  knowledge  and  performance  skills  are  quantified  using  oral  examinations  (e.  g.. 
Objective  Structured  Clinical  Examinations  [OSCE])  and  supervisor  ratings.  For 
example,  the  OSCE  was  developed  to  measure  clinical  competencies  for  both  medical 
students  and  residents  in  medical  diagnosis,  treatment  interventions,  laboratory 
implications,  and  follow-up  care. 
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Comprehensive  clinical  achievemenl  in  llie  United  States  is  assessed  using  the 
national  Ucensing  examinations,  NBME  Part  II  and  Part  111  until  1992,  and  thereafter, 
USMLE  Step  2  and  Step  3.  Step  2  licensing,  usually  completed  during  senior  year  of 
medical  school,  contains  over  700  multiple-choice  questions  which  assess  medical 
knowledge  and  an  understanding  of  clinical  science  essential  for  the  provision  of  patient 
care  under  supervision.  It  includes  an  emphasis  on  health  promotion  and  disease 
prevention.  This  examination  ensures  that  due  attention  is  devoted  to  the  principles  of 
clinical  sciences  that  reflect  the  safe  and  competent  practice  of  medicine.  Step  3 
licensing,  usually  completed  during  the  first  year  after  graduation  from  medical  school, 
assesses  whether  physicians  can  apply  medical  knowledge  and  an  understanding  of 
biomedical  and  clinical  science  essential  for  the  unsupervised  practice  of  medicine,  with 
emphasis  on  patient  management  in  ambulatory  settings.  This  examination  provides  a 
final  assessment  of  physicians  who  will  be  assuming  independent  responsibility  for 
delivering  general  medical  care. 

Similar  in  orientation  to  the  clinical  achievement  assessment  in  USMLE  Step  2 
and  Step  3,  licensing  in  Canada  includes  two  qualifying  examinations.  Medical  Council 
of  Canada  Qualifying  Examination  (MCCQE)  Part  I  (written  in  May  of  the  senior  year), 
and  Part  II  (written  after  1 6  months  of  post-graduate  training).  Locally  developed  written 
assessments  of  clinical  knowledge  include  multiple  choice  questions,  short  answer, 
and/or  essay.   When  hybrid  formats  are  employed  for  assessment,  they  are  referred  to  as 
multi-method  assessments. 

Problem-solving  achievement.  Is  operationally  defined  as  search  strategies, 
data-gathering,  accuracy  of  explanation,  coherence  of  explanation,  or  use  of  science 
concepts  linking  causes  and  hypothesis  (e.g.,  diagnostic  reasoning  and  accuracy).  The 
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initial  four  elements  defining  problem-solving  are  considered  important  because  they  are 
associated  with  expertise  and  differential  diagnosis  strategies  and  the  last  eleinent  is 
important  because  it  reflects  a  principal  aim  of  problem-based  teaching  and  learning. 

Self-directed  learning.  Is  operationally  defined  as  comprising  one  or  more  of  the 
following  elements  including  average  time  spent  in  independent  study  (e.g.,  medical 
texts,  journals),  medical  discussions  with  peers  or  mentors  outside  of  formal  settings, 
visits  to  the  library,  use  of  library  or  other  information  resources  (e.g.,  computer 
searches),  perceptions  of  self-directed  learning  and  espoused  approaches  to  study. 
Approach  to  self-study  emphasized  by  problem-based  instruction  includes  learning  for 
understanding  (meaning)  and  flexibility  rather  than  reproducing  information  (rote 
memorization). 

Attitude  toward  learning.  Is  operationally  defined  as  self-rated  student 
perceptions  regarding  various  aspects  of  the  learning  environment,  including  student 
perception  of  teaching,  the  congruence  between  stated  learning  objectives  and 
assessments,  effectiveness  of  pedagogical  methods,  preparedness  for  clinical  practice 
(e.g.,  utility  of  learning),  and  faculty  attitudes.  For  example,  the  Medical  School 
Learning  Environment  Survey  (MSLES  Form  A  and  B)  is  used  to  tap  this  construct.  The 
MSLES  consists  of  seven  subscales  including  nurturance,  organization,  flexibility, 
student-to-student  interaction,  breadth  of  interests,  emotional  climate,  and  meaningful 
learning  experience. 

Context  of  Study 

A  search  was  conducted  to  identify  primary  research,  both  published  and 
unpublished,  for  the  period  January  1966  through  January  2003  comparing  the  relative 
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efficacy  of  problem-based  versus  lecture-based  teaching  and  learning  in  medical  students 
and  physicians.  This  search  included  perusal  of  electronic  databases,  published  literature 
reviews,  bibliographies  of  primary  research,  and  the  examination  of  journals  likely  to 
publish  PBL  and  LBL  research.  PBL  databases  maintained  by  experts  in  the  field  also 
were  examined  to  insure  the  consideration  and  inclusion  of  all  relevant  research 
literature.  Reports  were  selected  if  results  were  appropriate  to  compute  a  standardized 
effect-size  comparing  knowledge  achievement,  clinical  achievement,  problem-solving, 
self-directed  learning,  or  attitudes  toward  learning  between  problem-based  and  lecture- 
based  instruction. 

Significance  of  Results 
This  is  the  first  study  to  systematically  aggregate  research  findings  and  analyzes 
statistically  the  variance  of  effect  size  estimates  to  examine  potential  moderators  of 
problem-based  teaching  and  learning  efficacy  in  medical  student  and  physician 
education.  Results  of  this  research  are  not  definitive  due  to  limitations  on  cause  and 
effect  inference.  However,  the  strength  of  this  meta-analysis  compared  to  primary 
research  resides  in  its  ability  to  investigate  data  accrued  over  muUiple  replications  using 
varied  operational  definitions;  thus  constructing  aggregate  estimates  of  magnitudes  of 
effect  sizes  for  theorized  relations.  The  present  research  facilitated  a  broader  perspective 
with  increased  statistical  power  for  clarifying  and  understanding  the  findings  of  primary 
research  comparing  problem-based  and  lecture-based  instruction.  Patterns  not  evident  in 
primary  research  emerged  and  weight  of  inferences  and  conclusions  have  unique  merit  of 
being  current  best  evidence-based. 
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The  superiority  of  traditional  perspectives  of  biomedical  science  knowledge 
transfer  (e.g.,  positivism)  is  not  supported  by  results  of  this  research.  The  findings  of  this 
research  suggest  that  responsibility  for  learning  can  be  transferred  to  students  resulting  in 
biomedical  science  achievement  at  least  equivalent  to,  and  clinical  science  achievement 
and  problem-solving  skills  superior  to  traditional  lecture-based  approaches  while 
facilitating  self-directed  learning  in  a  more  enjoyable  manner.  Despite  small  effect  sizes 
by  conventional  standards  (Cohen,  1988)  for  clinical  achievement  (ES  =  0.32)  and 
problem-solving  skills  (ES  =  0.19),  the  evidence  is  consistent  with  the  perspectives 
advanced  by  some  situational  learning  theorists  (Brown  et  al,  1989;  Greeno,  1998;  Lave 
&  Wenger,  1991);  and  the  problem-based  instructional  effect  size  magnitude  is  similar  to 
values  reported  in  other  similar  domains  (Lipsey  &  Wilson,  1993,  2001).  Furthennore,  it 
is  important  to  appreciate  the  fact  that  effect  size  cannot  be  accepted  literally.  This  is 
because  outcome,  methodological  and  substantive  variables  influence  the  magnitude  of 
effect  size.  Present  findings  suggest  that  even  in  the  same  medical  education  research 
domain,  not  all  effect  sizes  are  created  equally. 

Present  research  revealed  varied  operational  characteristics  of  problem-based 
teaching  and  learning.  The  functional  details  of  implementation  are  missing  from  much 
of  the  primary  research,  thus  discovering  essential  elements  and  perfecting  them  remain 
problems  for  future  process-oriented  research.  Although,  there  was  little  systematic 
examination  among  studies  to  measure  the  actual  differences  between  instructional 
methods,  an  important  implication  of  this  research  is  that  teaching  and  learning  with 
versus  without  the  integration  of  biomedical  and  clinical  science  facilitates  superior 
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clinical  science  achievement  and  problem-solving  skills  in  medical  students  and 
physicians. 

Despite  the  limitations  of  this  research  current  best  evidence  derived  trom  a 
synthesis  of  the  available  research  suggests  PBL  versus  LBL  affects  the  efficacious 
integration  of  medical  science  theory  and  practice  as  evidenced  by  superior  clinical 
science  knowledge  and  problem-solving  achievement  while  promoting  self-directed 
learning  and  positive  attitudes  toward  learning,  which  are  the  expressed  aims  of  medical 
education  (Liaison  Committee  on  Medical  Education,  2002). 

Limitations  of  Study 

The  data  employed  in  this  research  are  only  as  valid  as  the  primary  studies. 
Consequently,  answers  to  questions  posed  by  this  research  are  presented  to  the  extent 
available  in  the  literature.  Ultimately,  however,  findings  and  inferences  resulting  from 
this  research  were  influenced  primarily  by  the  methods  employed  to  analyze  these  data 
and  interpret  the  results  within  the  framework  of  principles  of  information  processing, 
social  constructivism  and  situated  learning.  Main  effects  for  clinical  science 
achievement,  problem-solving  success,  self-directed  learning  skills  and  positive  attitudes 
toward  learning  are  robust  and  can  be  generalized  with  significant  confidence,  but 
analyses  of  some  moderators  were  limited  by  small  data  sets.  Implementation  details  are 
missmg  from  much  research  literature,  which  limited  the  sensitivity  of  analyses,  and 
heterogeneity  within  categories  of  many  study  features  could  not  be  resolved,  which 
renders  impossible  unambiguous  inferences.  Indeed,  a  significant  limitation  of  the 
available  comparative  research  is  that  it  focuses  almost  exclusively  on  learning  outcomes 
rather  than  the  process  of  learning.  This  is  incompatible  with  recent  attention  afforded  to 


29 

the  principles  of  differential  psychology  and  the  social  construction  of  knowledge.  Thus, 
discovering  the  essential  elements  of  problem-based  teaching  and  learning  and  then 
perfecting  them  remains  for  future  research.  Expectantly,  future  research  will  be  guided 
by  the  specific  gaps  identified  and  examined  in  the  present  research. 


CHAPTER  2 
REVIEW  OF  THE  LITERATURE 

This  research  is  designed  to  examine  the  differential  efficacy  of  problem-based 
versus  lecture-based  teaching  and  learning  in  medical  education.  Theoretical  framework 
for  this  research  is  grounded  primarily  by  educational  philosophy,  information-processing 
theory,  and  principles  of  social  constructivism  and  situated  learning.  Throughout  the 
literature  review,  I  will  employ  this  framework  to  provide  insight  for  the  interpretation  of 
aims,  claims,  and  research  evidence  concerning  the  cognitive  and  non-cognitive  efficacy 
of  problem-based  medical  education.  In  particular,  I  will  review  the  literature  exploring 
the  development  of  medical  problem-solving  expertise,  and  the  roles  of  problems, 
students,  small-group  tutorial,  and  self-directed  learning  in  problem-based  teaching  and 
learning.  A  synthesis  of  findings  and  implications  leading  to  the  present  study  conclude 
the  chapter. 

Theoretical  and  Scientific  Foundations  of  Problem-Based  Medical  Education 
Problem  posing  as  an  instructional  method  has  been  employed  for  at  least  two- 
millennia  (Dewey,  1948).  Ancient  Greek  educators  challenged  students  with  problems  to 
stimulate  critical  inquiry,  problem  solving,  theory  construction,  reasoning,  and  the 
acquisition  of  knowledge.  Problem-based  teaching  and  learning  is  a  contemporary 
manifestation  of  this  classical  Greek  instructional  method.   It  was  introduced  to  remedy  a 
medical  school  curriculum  that  over-emphasized  rote  memorization,  artificially  isolated 
the  biomedical  sciences  from  clinical  training,  and  failed  adequately  to  prepare  medical 
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students  for  the  kinds  of  problems  encountered  in  clinical  practice  (Neufeld  &  Barrows, 
1974;  Barrows  &  Tamblyn,  1980).  As  mentioned  previously,  this  anecdotally  based 
concern  was  empirically  substantiated  by  a  sui-vey  of  453  senior  students  undergoing 
traditional  medical  education  at  five  universities  (Regan-Smith  et  al.,  1994).  Forty-nme 
percent  of  the  students  reported  they  learned  half  or  more  of  biomedical  science  by 
memorizing  without  understanding.  Rote  learning  does  not  provide  the  student  with  the 
basic  knowledge  necessary  for  solving  clinical  problems  (Bordage  &  Zacks,  1984).  In 
particular,  the  inability  to  solve  clinical  problems  is  due  both  to  a  lack  of  organization  and 
a  lack  in  understanding  the  relevant  information  stored  in  memory  (Bordage  &  Zacks; 
Sobral,  1995).  The  successful  resolution  of  complex  medical  problems  is  hypothesized  to 
most  likely  occur  when  medical  information  is  linked  to  networks  of  concepts  and  their 
relationships,  then  compiled  into  compact  representations  called  "chunks'"  (Anderson, 
PiroUi,  &  FaiTel,  1988;  Chase  &  Simon,  1973)  or  "schemata"  (Perkins  &  Salomon,  1989) 
that  links  medical  knowledge,  clinical  procedural  knowledge,  and  production  rules 
(Sobral). 

Sobral  (1995)  confirmed  this  hypothesis  in  medical  students  employing  the 
validated  and  reliable  Diagnostic  Thinking  Inventory  (Bordage,  Grant,  &  Marsden, 
1990).  The  Inventory  measures  flexibility  in  thinking  and  knowledge  stnictures  in 
memory  (Bordage  et  al).  Sobral  studied  the  relationship  between  knowledge 
representation  and  the  application  of  knowledge  to  problem  solving  in  third-year  medical 
students  undergoing  problem-based  versus  lecture-based  learning.  He  discovered 
students  undergoing  PBL  exhibited  significantly  higher  scores  in  knowledge  structures  in 
memory  measure  (ES  =  0.37,  2=01 8),  but  statistically  equivalent  flexibility  in  thinking 
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scores  (ES  =  0. 1 1 ,  g  =  .52).   Intergroup  disparity  in  functional  knowledge  structures  was 
independent  of  academic  achievement,  gender,  and  validated  and  reliable  measures  of 
learning  style,  self-confidence,  and  motivation  to  learn.  Detailed  findings  and  analyses 
are  presented  in  Chapter  4.  Sobral  concluded  that  the  difference  in  diagnostic  expertise 
was  due  to  increased  organization  and  understanding  of  information,  an  observation  that 
confimis  and  extends  the  findings  of  other  medical  (Boshuizcn,  Schmidt  &  Wassmer, 
1994;  Nomian  &  Schmidt,  1992)  and  nonmedical  (Chase  &  Simon,  1973;  Chi, 
Feltovitch,  &  Glasser,  1981)  educators. 

In  a  classic  study.  Chase  and  Simon  (1973)  asked  expert  and  novice  chess  players 
to  view  for  5-seconds  to  10-seconds  chessboards  arranged  with  authendc  games,  then  to 
recall  the  positions  of  the  25-chess  pieces.  Investigators  observed  expert  chess  players 
could  accurately  position  90%  of  the  chess-pieces,  whereas  novices  achieved  only  20% 
accuracy.  Accounfing  for  this  differential  accuracy,  the  authors  concluded  that  the  long- 
term  memory  of  chess  experts  contains  many  patterns  associated  with  successful  chess 
maneuvers  and  strategies  (i.e.,  pattern  recognition).  However,  expert  and  novice 
accuracy  were  equivalent  when  trials  were  repeated  with  randomly  positioned  chess- 
pieces.  Apparently,  unless  patterns  of  chess  pieces  emerge  from  natural  play  (i.e., 
typical),  advantage  of  expertise  vanishes.  That  is,  in  an  inauthentic,  decontextualized 
situation  (i.e.,  atypical),  expertise  is  significantly  diminished,  even  in  an  expert's  domain. 
Implications  for  education  are  that  although  while  domain-specific  knowledge  (e.g., 
biomedical  science  knowledge)  is  essential,  complex  problems  can  not  be  solved  by 
retrieving  information  and  applying  procedural  rules.  Since  the  resolufion  of  complex 
problems  depends  upon  context,  domain-specific  knowledge  must  be  integrated 
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appropriately  to  fit  in  tlie  problem-space.  That  is,  goodness-of-fit  between  context- 
indexed  knowledge  and  problem-context  influences  the  accuracy  of  problem  solving 
perfonnance  (Perkins  &  Salomon,  1989). 

In  a  similar  study,  Chi  and  associates  (1981)  experimentally  compared  expert 
physicists'  and  physics  students'  understanding  of  physics  problems  by  asking  them  first 
to  categorize  word  problems,  and  then  to  classify  isomorphic  problems  into  similar 
categories.  Investigators  discovered  novices  focused  upon  superficial  features,  whereas 
experts  articulated  problems  according  to  the  types  of  physical  principles,  theorems,  and 
laws  required  to  resolve  the  problem.  Importantly,  experts'  mental  representations 
appear  to  be  grounded  in  solution  principles  (e.g.,  solved  exemplars).  Finding  suggests 
that  experts  have  access  to  vast  procedural  knowledge  about  solutions  and  their 
applicability.  This  allows  experts  to  represent  problems  on  a  level  directly  related  to  their 
solution  or  improvement.  Compared  to  experts,  novices  have  less  access  to  information 
about  effective  solution  procedures,  a  view  expanded  by  social  constructivists  (e.g., 
pragmatic  constructivism,  Cobb,  2002)  and  situational  learning  theorists  (e.g., 
concurrently  linking  concepts  and  context-particular  solutions.  Brown  et  al.,  1989; 
Perkins,  1986),  and  is  discussed  later. 

Taken  together,  these  studies  suggest  that  the  ability  to  solve  medical  problems 
can  be  enhanced  best  by  initially  fostering  the  development  of  explanatory  mental 
representations  during  declarative  and  procedural  knowledge  acquisition.  Students  then 
subsequently  employ  the  schemes  to  recall  the  infomiation  needed  to  improve  or  solve  a 
clinical  problem  situation.   Encapsulating  both  knowledge  acquisition  and  problem 
solving  has  the  advantage  of  amalgamating  into  a  single  operation  knowledge 
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construction  and  a  search-and  retrieval  strategy  (Brown  et  al.,  1989;  Chi  et  al.,  1981; 
Lesgold  et  al.,  1988;  Perkins,  1986). 

Extending  their  initial  observations,  Chi,  DeLeeuw,  Chiu,  and  LaVancher  (1994) 
revealed  significant  disparity  between  good  and  poor  learners  in  the  extensiveness  and 
quality  of  problem  representation  and  self-explanations  advanced  as  students  negotiated  a 
problem-space.  Their  results  suggest  good  learners  also  exhibit  better  cognitive 
monitoring  (metacognition)  skills  and  the  ability  to  assess  what  they  have  learned  when 
challenged  with  a  novel  problem.  Self-explanations,  or  private  verbal  elaborations 
related  to  the  problem  challenge  are  viewed  by  the  authors  as  a  mode  of  studying  that  can 
mediate  learning.  Moreover,  these  observations  provide  evidence  for  the  active,  self- 
construction  of  knowledge. 

Vygotsky  (1926/1997;  1962)  described  a  similar  self-mediated  learning  strategy 
as  "private  talk."  In  a  similar  vein,  Gal'perin  (1969)  conceptualized  this  strategy  as 
"dialogical  thinking."  Dialogical  thinking  initially  involves  speaking  aloud  about  the  task 
or  concept,  followed  by  covert  speaking,  that  is,  "speech  minus  sound"  (p.  263).  In  the 
mind,  Gal'perin  argues,  overt  speech  is  represented  by  a  deep  structure,  or  as  he 
describes,  "the  audible  image  of  a  word"  (p.  263).  For  Gal'perin,  an  audible  image  is 
more  stable  and  durable  than  a  visual  image.  This  line  of  reasoning  is  similar  to  the 
theory  of  multiple  encoding  proposed  by  Paivio  (1986).  According  to  Paivio's  Dual 
Coding  Theory,  cognition  incoiporates  representational  systems  called  verbal  and  visual 
systems.  Accordingly,  when  information  is  coded  in  both  systems,  memory  is  enhanced. 
Medical  learning  from  this  perspective  would  not  only  focus  on  the  acquisition  of 
biomedical  and  clinical  science  knowledge,  but  also  the  cognitive  representation  of  this 
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knowledge.  Therefore,  efficiency  in  representing  medical  biowledge  for  appropriate  re- 
deployment in  clinical  practice  is  hypothesized  to  be  an  essential  ingredient  in  the 
development  of  problem-solving  expertise  (Boshuizen  et  al.,  1994;  Patel  et  al.,  1988; 
Schmidt,  Norman,  &  Boshuizen,  1990). 

Patel  and  colleagues  (1988)  examined  problem-solving  expertise  development  in 
medical  students.  They  asked  students  at  three  separate  stages  of  a  biomedical  sciences 
curriculum-novices,  in  their  first  six-months  of  study,  intemiediates  during  their  senior 
year,  and  relative  experts  after  approximately  two-years  of  clinical  experience—  to  solve  a 
clinical  problem  and  then  integrate  passages  of  relevant  biomedical  science  information 
into  their  pathophysiologic  explanations  of  the  problem  (causal  explanations). 
Intemiediate  students  were  observed  to  employ  more  biomedical  science  information  in 
their  problem-solving  explanations  than  either  novice  or  experts.  Explaining  the 
differential  use  of  biomedical  science  knowledge,  authors  presented  an  intuitively  logical, 
but  previously  unsubstantiated  assertion  that  novices  lacked  adequate  biomedical 
knowledge,  and,  unlike  intermediate  students,  experts  filtered  out  inessential  biomedical 
information  in  their  reasoning  process.  The  implications  for  medical  education  are 
important,  as  these  findings  suggest  that  there  is  a  necessary  threshold  for  study  in  the 
biomedical  sciences  if  students  are  to  achieve  sufficient  problem  solving  expertise. 

Boshuizen  and  associates  (1994)  conducted  an  investigation  comparing 
instructional  infiuence  on  the  integration  of  biomedical  and  clinical  sciences  for  problem 
solving.  They  challenged  students  undergoing  either  pre-clinical  education  with 
problem-based  or  lecture-based  instructional  curriculum  with  an  identical  metabolic 
problem.  Students  were  asked  to  solve  the  problem  employing  their  knowledge  of 
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biomedical  science  (biociiemistry)  and  clinical  science  (internal  medicine).  Students 
from  problem-based  instructional  backgrounds  initially  employed  an  analytical  approach 
to  exploring  biochemical  aspects  and  subsequently  linked  them  to  clinical  elements  of  the 
problem  (i.e.,  causal  reasoning).  In  contrast,  students  from  lecture-based  instructional 
backgrounds  tended  to  employ  a  memory-based  approach  to  ascertain  a  solution  directly, 
a  common  process  when  learning  occurs  via  rote-memorization.  Similar  observations  in 
medical  students  undergoing  traditional  lecture-based  teaching  and  learning  have  been 
reported  by  other  investigators  (Lindblom-Ylanne  &  Lonka,  1999,  2002;  Miller,  1961 ; 
Neame,  1984;  Regan-Smith  et  al.,  1994).  Inquiry  strategy  employed  by  students 
undergoing  problem-based  teaching  and  learning  resulted  in  significantly  greater  problem 
solving  accuracy.  Detailed  results  and  analyses  are  presented  in  Chapter  4. 

Schmidt  and  colleagues  (1996)  conducted  a  similar  study,  but  with  thirty 
diagnostic  problems  epidemiologically  representative  of  Dutch  medical  practice.  Also 
included  in  the  study  were  students  attending  a  medical  school  employing  an  integrated 
(didactics  integrating  biomedical  and  clinical  sciences),  teacher-directed  curriculum. 
Compared  to  students  undergoing  typical  lecture-based  instruction,  students  undergoing 
integrated  lecture-based  and  reiterative  problem-based  instruction  exhibited  superior 
diagnostic  accuracy.  There  were  no  differences  in  diagnostic  accuracy  between  students 
from  integrated  and  PBL  ctirricula.  Detailed  results  and  analyses  are  presented  in 
Chapter  4.  The  authors  suggest  that  the  integration  of  biomedical  and  clinical  sciences 
grounded  in  patient  problems  is  likely  an  essential  feature  that  enables  superior  diagnostic 
perfonnance  independent  of  whether  instruction  is  teacher-  or  student-centered.  This 
observation,  if  coiToborated,  has  significant  implications  for  medical  education  and 
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provides  support  for  teaching  medical  concepts  as  tools  (Brown  et  al.,  1989;  Perkins, 
1986),  that  is,  the  integration  of  theory  and  practice,  or  pragmatic  constructivism  (Cobb, 
2002).  which  is  discussed  later. 

SchiTiidt  and  associates  (1990)  propose  that  medical  students  transition  from 
concept  novice  to  content  expert  in  four-stages.  Stage  one  is  characterized  by  the 
formation  of  elaborate  causal  networks.  These  networks  account  for  the  causes  and 
consequences  of  patient  problems  using  underlying  pathophysiological  mechanisms. 
Practice  fosters  the  apphcation  of  declarative  knowledge  and  networks  progressively 
compile  into  "high-level,  simplified  causal  models  explaining  signs  and  symptoms  and 
are  subsumed  under  diagnostic  label"  (Schmidt  et  al.,  p.  614).  Transition  to  a  second 
hypothesized  developmental  stage  occurs  when  medical  students  undergo  exposure  to 
patient  diagnosis  challenges  employing  a  process  called  "differential  diagnosis."  For  the 
novice,  "differential  diagnosis"  requires  significant  cognitive  effort  during  which  students 
attempt  to  link  patient  symptoms  (problem-space  cues)  to  knowledge  in  a  relevant 
pathophysiological  network.  Presumably,  continued  clinical  practice  results  in  the 
development  of  abridgments  (the  elimination  of  sub-goals)  and  the  encapsulation  of 
knowledge  (compact  representation)  corresponding  to  the  process  of  "chunking" 
described  in  information  processing  literature  (Anderson,  1990).  Anderson  introduced 
this  idea  in  psychology  to  explain  the  learning  by  doing  phenomenon.  By  reconstructing 
representation  of  medical  information,  knowledge  can  be  structured  and  organized  into  a 
fomi  that  facilitates  faster  deployment  (Bordage&Zacks,  1984;  Glasser,  1984,  1991; 
Perkins,  1986).  For  Anderson,  the  basic  idea  behind  differential  construction  of  cognitive 
representations  for  deployment  efficiency  is  to  remove  knowledge  that  is  only  employed 
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in  the  intemiediate  steps.  Theoretically,  reconstructed  knowledge  streamlines  or  speeds 
up  processing  which  allows  problem-solution  to  be  reached  directly  from  problem-space 
cues.  Thus,  as  successful  differential  diagnosis  experience  increases,  medical  students 
become  less  likely  to  activate  all  possible  relevant  knowledge  to  learn  diagnosis;  rather 
only  the  knowledge  relevant  for  a  particular  case  is  activated. 

Script  theory  suggests  that  in  given  situations  students  activate  knowledge  sets  to 
understand  a  situation.  For  Schmidt  and  colleagues  (1990),  this  ability  to  distinguish  the 
particulars  of  a  case  without  extensive  knowledge  activation  represents  a  transition  from 
a  network  model  to  an  illness-scripts  model  (Stage  3).  Illness-scnpts  only  compile 
information  required  for  diagnosis  and  treatment,  e.g.,  antecedent  conditions,  patient- 
problem  description,  signs,  and  symptoms.  Scripts  contain  data  about  links  that  unite 
items  of  knowledge  (e.g.,  clinical  cues  or  features)  related  to  an  illness.  In  problem- 
situations,  these  links  facilitate  decisions  regarding  the  strengths  and  weaknesses  of  a 
tentative  diagnosis. 

Stage  four  of  clinical  problem-solving  expertise  development  occurs  when  illness- 
scripts  remain  available  in  long-temi  memory  to  be  employed  for  use  subsequently  when 
similar  patient  problems  present.  That  is,  the  memory  of  specific  patients  is  represented 
as  "individual  entities  and  not  merged  into  some  prototypical  fomi"  (p.  617). 
Consequently,  clinical  reasoning  is  fueled  by  similarity  in  presenting  cases  and  the 
retrieved  memory  of  previous  patient(s).  The  authors  suggest  that  these  memories  are 
stored  in  episodic  memory  rendering  them  readily  retrievable. 

In  summary,  illness  scripts  theory  postulates  that  physicians  consistently  recall 
information  in  the  same  sequence  independently  of  how  data  were  presented. 
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Accordingly,  when  confronted  with  a  patient  problem,  a  physician  searches  his  repertoire 
of  illness-scripts  for  a  match  (pattern  recognition),  which  further  instantiates  relevant 
illness-script.  As  discussed  earlier,  similar  cognitive  processes  were  observed  in  expert 
chess  players  (Chase  &  Simon,  1973)  and  physicists  (Chi  el  al.,  1981). 

An  unanswered  question  is  what  biomedical  science  content  threshold  must  be 
attained  before  students  can  self-explain  (Chi  et  al.,  1989,  1994;  Gal'perin,  1969; 
Vygotsky,  1926/1997;  1962),  monitor  their  learning  (Semb  &  Ellis,  1994;  Winne,  1995a, 
1995b),  and  effect  integration  of  declarative  knowledge  (biomedical  sciences)  and 
procedural  knowledge  (clinical  sciences)  to  facilitate  the  development  of  production  rules 
(Anderson,  1990)  or  differential  diagnosis  skills  (Schmidt  et  al.,  1996)?  Wirme  suggests 
the  quality  of  self-regulated  learning,  a  key  feature  of  reiterative  PBL  (although  called 
self-directed  learning  by  medical  educators)  may  be  affected  by  a  student's  domain 
specific  declarative  knowledge.  If  a  learner's  mental  representations  for  a  topic  are 
limited  (i.e.,  low-domain  knowledge),  there  is  less  ability  to  associate  novel  infomiation 
(Schon,  1987).  This  is  true  for  any  fonn  of  cognitive  learning  from  recall  of  declarative 
knowledge  to  application  of  concepts  or  production  rules  (Anderson,  1990;  Schon).  This 
question  addresses  instructional  design  concerns  regarding  the  magnitude  of  the 
biomedical  science  foundation  necessary  before  medical  students  can  efficaciously 
engage  in  solving  clinical  problems  (e.g.,  differential  diagnosis)  and  self-direct  their 
learning  performance.  This  will  be  discussed  later. 

As  mentioned  previously,  Flexner  (1910,  1925)  addressed  the  tension  between 
acquiring  knowledge  and  applying  it  by  insisting  a  solid  foundation  in  biomedical  science 
was  a  prerequisite  for  subsequent  clinical  encounters.  Positing  an  extreme  position. 
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Flexner  opposed  any  clinical  education  during  the  first  two-years  of  medical  school 
(1925).  Consequently,  as  was  discussed  in  Chapter  1,  traditional  medical  school 
instruction  is  characterized  by  the  temporal  isolation  of  biomedical  and  clinical  sciences 
(Ebert,  1992;  Schon,  1987).  Whereas,  as  discussed  earlier  William  Osier  (1903)  argued 
for  a  more  practice-based  medical  education.  Reflecting  epistemology  advanced  by 
Osier,  Neufeld  and  Barrows  (1974)  believe  imparting  biomedical  science  independent  of 
the  clinical  context  in  which  it  needs  to  be  applied  is  inappropriate.  Neufeld  and  Barrows 
view  clinical  problems  as  contextual  focal  points  for  integrating  biomedical  and  clinical 
sciences  into  a  coherent  and  practical  learning  experience.  As  a  result,  problem-based 
learning  was  conceived  of  as  an  instructional  method  employing  patient  problems  as  the 
context  for  acquiring  knowledge  about  biomedical  and  clinical  science  and  transitioning 
from  content  novice  to  content  expert  while  cultivating  metacognitive  and  problem- 
solving  skills  (Barrows,  1986,  1990,  1996,  2000;  Barrows  &  Tamblyn,  1980;  Boud  & 
Feletti,  1997;  Neufeld  &  Barrows).  Problem-based  learning  emerged  from  an 
educational  stance  that  envisioned  learning  as  central  to  the  educational  process,  and 
learners  as  active,  self-directed  participants  in  that  process  (Neufeld  &  Barrows).  That  is, 
problem-based  teaching  and  learning  is  not  solely  the  addition  of  problem-solving 
activities  to  an  otherwise  discipline-centered  curricula;  problem-based  learning  is  an 
educational  method  applied  to  a  domain  of  knowledge  rather  than  an  isolated 
instructional  technique  (Engel,  1997). 

A  pedagogical  strategy  incorporating  problem-based  learning  is  aligned  with 
Glasser's  (1991)  view  that  the  psychology  of  instruction  should  strive  to  understand  the 
development  of  cognitive  structures  and  the  processes  that  characterize  the  competent 
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performance  of  cognitive  skills  by  experts  in  particular  knowledge  domains.  In  concert 
with  Schon  (1987)  and  Perkins  (1986),  Glasser  argues  that  instructional  design  is  best 
when  guided  by  a  theoretical  framework  characterizing  the  acquisition,  structure  and 
organization  of  knowledge  guiding  expert  practitioners.  In  contrast,  traditional  lecture- 
based  instructional  approaches  first  present  discipline-oriented  biomedical  science,  and 
then  clinical  science,  during  which  students  presumably  learn  to  apply  this  knowledge  to 
health  care  problems  of  everyday  practice.  Assumed  apperception  may  be  more  apparent 
than  real  (Schon).  Because  research  suggests  much  of  biomedical  science  learned  in 
traditional  manner  subsequently  is  unable  to  be  recalled,  or  discovered  to  be  irrelevant 
(Miller,  1961;  Neame,  1984;  Schmidt,  1983;  Schon). 

More  than  40-years  ago.  Miller  (1961)  observed  poor  retention  of  biomedical 
science  information  by  his  medical  students  who  undei-went  lecture-based,  discipline- 
oriented  instruction  and  warned  that  information  presented  via  teacher-centered  methods 
would  "decrease  from  memory  at  the  same  rate  as  nonsense  syllables"  (p.  153).  Further 
evidence  is  provided  by  Levine  and  Fonnan  (1973)  who  presented  fifty  clinically  relevant 
questions  to  students  starting  a  neurology  clerkship  (senior  year).  Questions  were 
obtained  from  a  final  examination  each  student  had  passed  in  their  first  year  of  medical 
school.  Upon  retesting,  only  48%  of  senior  medical  students  received  a  passing  grade. 
The  authors  concluded,  "a  substantial  number  of  students  fail  to  retain  the  information 
taught  in  the  preclinical  years"  (p.  869).   Such  results  prompted  Neame  (1984)  to  write 
"there  is  little  evidence  that  the  existing  preclinical  courses  are  essential  to  high  quality 
medical  practice;  mastery  of  them  seems  to  confer  little  benefit"  (p.  707). 
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These  outcomes  may  be  attributable  to  the  intentional  demarcation  between 
theoretical  and  clinical  sequencing  of  conventional  medical  curricula  (Schon,  1987). 
Medical  students  educated  in  this  manner  encounter  difficulty  when  they  transition  from 
theory  to  practice  (Patel  et  al.,  1 989).  They  are  inadequately  prepared  when  they 
encounter  problems  requiring  the  transfer  of  their  learning  to  new  domains,  an  ability 
essential  to  functioning  effectively  in  practice  (Ban-ows,  1986,  1990,  1996,  2000;  Schon). 
These  findings  have  important  implications  for  medical  education  in  general  and  clinical 
practice  competence  in  particular;  to  manage  the  relentless  medical  knowledge  explosion 
efficiently,  students  and  physicians  must  have  an  extensive  awareness  of  sciences  basic  to 
medicine  (Barrows,  1983;  Tosteson,  1994). 

Barrows  ( 1 983,  2000)  posits  that  when  biomedical  science  content  is  coupled  with 

a  genuine  health  care  problem,  students  will  integrate  knowledge  into  meaningful 

representations.  This  epistemological  stance  is  nicely  characterized  by  Glasser  (1991) 

who  wrote  "Instruction  should  have  the  mission  of  making  itself  unnecessary;  learners 

should  become  mindful  architects  of  their  own  knowledge"  (p.  131).  Indeed,  the 

philosophy  of  PBL  is  represented  in  an  ancient  Chinese  proverb  (Smith,  1984,  cited  by 

Bickley,  Donner,  Walker,  &  Tift,  1990). 

If  you  tell  me,  I  will  forget. 

If  you  show  me,  I  might  remember. 

But  if  you  involve  me,  I  will  learn. 

Describing  the  etyinology  of  the  word  cumculum,  Walton  and  Mathews  (1989) 

emphasized  a  curriculum  is  more  than  a  matrix  of  topics,  just  as  a  house  is  more  than  an 

assemblage  of  bricks  and  mortar.  For  Walton  and  Mathews,  the  "bonding"  and 

"structure"  are  essential  features  of  curricular  construction.  Theoretically,  a  medical 
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curriculum  that  promotes  contextual  learning  and  self-directed  learning  and  fosters 
motivation  while  focusing  on  the  health  care  needs  of  society  provides  substantial 
bonding  and  structure.  Problem-based  instruction  appears  to  support  this  approach  more 
adequately  than  discipline-oriented,  teacher-centered,  syllabus-driven,  lecture-based 
instruction.  Problem-based  instruction  aims  to  (a)  structure  knowledge  for  use  in  clinical 
contexts,  (b)  develop  effective  clinical  reasoning  process,  (c)  develop  effective  self- 
directed  learning  skills,  and  (d)  increase  a  positive  attitude  toward  learning  (Barrows, 
1986).  Reverberating  principles  of  adult  learning  theory  (Knowles,  1975),  Barrows 
suggested  the  nature  of  health  care  problems  and  the  locus  of  control  of  learning  effect 
the  probability  of  realizing  the  aims  of  PBL.  The  nature  of  health  care  problems  can  be  a 
complete  case,  a  vignette,  or  a  comprehensive  problem  simulation.  Locus  of  learning  can 
be  teacher-centered,  student-centered,  or  a  hybrid  of  the  two.  Consequently,  there  are 
varied  instructional  approaches  to  problem-based  teaching  and  learning. 

As  discussed  in  Chapter  1 ,  Barrows  ( 1 986)  developed  a  taxonomy  to  distinguish 
operational  variants  of  problem-based  learning.  For  Barrows,  only  reiterative  PBL 
achieves  all  four  aims,  thus  this  pedagogical  classification  is  "as  much  a  taxonomy  of 
teaching-learning  methods,  within  which  problem-based  learning  fits,  as  it  is  of  problem- 
based  learning  itself  (p.  38).  Reiterative  PBL  views  construction  of  a  framework  of 
knowledge  and  ways  of  thinking  about  clinical  problems  as  essential  features  of  effective 
medical  education.  Moreover,  quintessential  elements  of  the  reiterative  PBL  framework 
is  that  it  includes  all  antecedents  of  health  and  expresses  the  Aristotelian  aphorism  that 
"medicine  begins  in  philosophy,  and  philosophy  ends  in  medicine"  (cited  in  Tosteson, 
1994,  p.  9).  From  this  perspective,  PBL  appears  to  be  both  a  pedagogical  method  and  a 
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philosophy  of  education  at  once  (Maudsley,  1999).  A  reiterative  PBL  cumculum  is 
structured  to  foster  a  process  designed  to  establish  and  sustain  an  environment  of  free- 
inquiry  and  co-learning  which  includes  a  tutored  group  of  students  (Barrows,  1990). 

Free-inquiry  and  collaboration  provides  an  opportunity  for  the  kind  of  academic 
pluralism  espoused  by  Dewey  (1933/1998).  This  process  emerges  in  a  particular 
sequence,  for  example,  the  Maastricht  University  (The  Netherlands)  "seven  jump" 
(Schmidt,  1983),  which  enables  learners  to  identify  their  needs  in  understanding  a 
problem  and,  once  these  are  identified,  to  pursue  their  goals  usually  independently,  and 
finally  regroup  to  synthesize  their  findings.  According  to  Schmidt,  the  process  should 
incorporate  the  following  seven-steps  (p.  12). 

1.  Tutor  clarifies  terms  and  concepts  not  readily  comprehensible; 

2.  Tutorial  group  defines  problems;  agree  which  phenomena  need  explanation; 

3.  Tutorial  group  analyzes  problem  via  brainstorming  based  on  prior  knowledge  and 
intuition; 

4.  Tutorial  group  proposes  possible  explanations  and  develops  working  hypotheses; 

5.  Tutorial  group  generates  and  prioritizes  learning  issues; 

6.  Students  research  learning  issues  via  self-directed  learning; 

7.  Students  synthesize  explanations  in  tutorial  group,  and  collaboratively  apply 
newly  acquired  information  to  the  problem. 

Savery  and  Duffy  (1995)  contribute  additional  insight  with  a  comprehensive 

model  for  reiterative  PBL  implementation  incorporating  principles  of  social 

constructivism  and  ways  to  apply  those  principles  to  learning  and  understanding.  They 

characterize  the  philosophical  stance  of  social  constructivism  with  three  propositions. 

First,  understanding  is  intertwined  with  one's  interactions  with  the  environment.   Second, 

cognitive  conflict  or  an  indeterminate  situation  is  a  stimulus  for  learning  and  influences 

the  organization  and  nature  of  what  is  learned.  Third,  knowledge  and  warranted 

assertions  emerge  via  social  interaction  and  self-assessment  of  understanding.  For 
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Savery  and  Duffy,  PBL  is  a  pedagogical  approach  tliat  most  ideally  reflects  and  employs 
these  propositions.  Grounded  in  this  framewortc,  Savery  and  Duffy  propose  eight 
instructional  principles  to  guide  the  implementation  of  PBL  (pp.  32-34). 

1.  Anchor  all  learning  activities  to  a  larger  problem. 

2.  Support  learner  in  developing  ownership  for  the  overall  problem  or  task. 

3.  Design  an  authentic  task. 

4.  Design  task  and  learning  environment  to  reflect  the  comple.xity  of  the 
environment  the  learner  should  be  able  to  function  in  at  end  of  learning. 

5.  Give  learner  ownership  of  process  used  to  generate  solution. 

6.  Design  learner  environment  to  support  and  challenge  learner's  thinking. 

7.  Encourage  testing  ideas  against  alternative  views  and  alternate  contexts. 

8.  Provide  opportunity  for,  and  scaffold  reflection  on  both  content  learned  and 
learning  process. 

What  then  are  the  active  ingredients  of  problem-based  teaching  and  learning? 
Shulman  (1986)  suggests  "the  curriculum  and  its  associated  materials  are  the  materia 
medica  of  pedagogy,  \\ie  pharmacopeia  from  which  the  teacher  draws  those  tools  of 
teaching  that  present  or  exemplify  particular  content  and  remediate  or  evaluate  the 
adequacy  of  student  accomplishments"  (p.  10).  Ban-ows  (1986),  Schmidt  (1983),  and 
Savery  and  Duffy  (1995)  provide  the  materia  medica  of  a  pedagogical  format  whereby 
medical  students  actively  acquire  infomiation  and  are  encouraged  to  reason  and  generate 
hypotheses  thai  can  subsequently  be  confirmed  or  refuted.  During  this  process,  students 
identify  learning  issues  to  be  studied  during  subsequent  periods  of  self-directed  learning, 
and  this  acquired  knowledge  is  shared  and  collectively  applied  to  analyze  authentic 
problems.  This  is  a  learning  process  characterized  by  free  inquiry,  collaboration  and  self- 
reliance,  l\\£  pharmacopeia  from  which  students  draw  tools  for  developing  expertise  and 
professional  practice  competence. 
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Features  of  Problem-Based  Learning 

Lave  and  Wenger  (1991)  differentiate  between  a  teaching  curriculum  and  learning 
curriculum.  "A  teaching  curriculum  provides  structured  resources  and  meaning  for  the 
learners  which  are  moderated  by  the  instructor."  Whereas,  a  learning  curriculum 
incorporates  situated  opportunities,  for  instance,  varied  exemplars  from  everyday  practice 
"viewed  from  the  perspective  of  the  learners"  (Lave  &  Wenger,  p.  97).   Engendering  this 
stance,  authentic  problem-centered  learning  builds  a  cognitive  repertoire  of  exemplars 
(e.g.,  schemata  or  illness  scripts)  from  which  knowledge  and  skill  may  be  appropriately 
retrieved  and  applied  to  the  solution  of  practice  problems  (Nonnan  &  Schmidt,  1992). 
Echoing  Dewey  (1933/1998),  this  stance  is  contrary  to  traditional,  or  objectivist  theories 
of  teaching  and  learning  which  value  abstract  knowledge  over  practice,  thus 
dichotomizing  theory  and  practice  (Biggs,  1996;  Schon,  1983,  1987). 

Problem-based  instruction  aims  to  focus  the  learning  process,  not  the  content.  In 
this  manner,  problem-based  instruction  transfers  control  of  the  learning  process  from 
teacher  to  student,  which  is  conforms  with  concepts  of  "learning  curriculum"  (Lave  & 
Wenger,  1991)  and  "thinking  curriculum"  (Resnick  &  Resnick,  1992,  p.  37).  As 
suggested  by  Barrows  (1986,  2000),  Schmidt  (1983),  and  Savery  and  Duffy  (1995), 
realizing  the  aims  of  problem-based  teaching  and  learning  requires  a  multidimensional 
instructional  strategy  that  incorporates  roles  of  problems,  students,  small-grotip  tutorials, 
self-directed  learning,  and  assessments. 
Role  of  Problems 

A  design  feature  of  problem-based  teaching  and  learning  is  to  construct  problems 
that  are  effective  in  fostering  an  adaptation  to  clinical  problem  space  and  cultivating 
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expertise.  In  this  respect,  problems  serve  as  tine  initial  stimulus  and  framework  for 
learning  biomedical  and  clinical  science  content.  Thus,  problems  "become  the  'cognitive 
scaffolding'  for  information  rather  than  the  scientific  discipline  itself  (Bickley,  1993,  p. 
545).  By  this  means,  problems  are  not  constrained  by  disciplines  and  designed  to 
integrate  biomedical  and  clinical  sciences.  From  an  educational  perspective,  "problem 
posing"  is  a  pedagogical  tool  employed  as  a  challenge  leading  to  learning  which  emerges 
while  working  on  the  problem.  A  problem  provides  a  context  that  theoretically  reinforces 
factual  learning  and  the  application  of  concepts.  Problems  can  be  employed  as  orienting 
activities  to  stimulate  curiosity,  illustrate  principles,  and  as  advanced  organizers  to 
stimulate  prior  knowledge  (Ausubel,  1960).  Consequently,  a  fundamental  assumption  of 
problem-based  instruction  is  that  it  renders  teaming  more  relevant,  and  thus  more 
transferable,  by  engaging  students  in  authentic  practice  problems  (Schmidt,  1993).  For 
Schmidt,  at  least  four-aiins  of  student  learning  are  served  by  employing  problems 
including  (a)  problems  trigger  the  activation  of  whatever  knowledge  is  available;  (b) 
authentic  problems  situate  activated  knowledge  and  novel  information  in  a  professional 
practice  context;  (c)  problems  trigger  cognitive  conflict  and  group  discussion  helps 
learners  to  elaborate  their  knowledge  structures;  (d)  problems  engage  students  in  such  a 
meaningful  manner  as  to  inspire  epistemic  curiosity  (the  desire  to  know). 

Problem  structure.  Problems  may  be  either  well-structured  or  ill-structured 
(Newell  &  Simon,  1972).   Well-structured  problems  are  those  with  a  single,  agreed  upon 
solution.  Both  initial  state  and  appropriate  end-point  (goal  state)  are  clear  and 
unambiguous.  Ill-structured  problems  are  those  without  an  agreed  upon  solution  and 
there  are  likely  to  be  differential  means  of  solving  the  problem.   However,  well- 
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structured  and  ill-structured  problems  should  not  be  viewed  as  a  dichotomy;  rather  they 
anchor  two  ends  of  a  problem-space  continuum  (Reitman,  1965).  Indeed,  as  Simon 
(1973)  noted,  many  authentic  problems  are  presented  as  ill-defmed  problems,  but  become 
well-structured  in  the  hands  of  the  problem  solver.  Simon  emphasizes  information 
provided  by  the  problem-solver  is  responsible  for  converting  an  ill-structured  problem  to 
a  well-structured  problem.  Simon  also  suggested  differential  content  knowledge  to 
discriminate  strong  from  weak  problem-solving  perfonnance,  but  cautioned  that  problem- 
solving  expertise  is  not  grounded  entirely  in  mastery  of  passively  recalled  content. 
Knowledge  must  be  retrieved  and  structured  to  be  functional  for  clinical  problem  solving 
(Bordage  &  Zacks,  1984;  Glasser,  1984,  1991;  Perkins,  1986;  Sternberg,  1994);  because 
medical  problems  often  necessitate  acquisition,  analysis,  and  synthesis  of  supplemental 
information  (Elstein,  Shulman,  &  Spafka,  1978). 

The  initial  step  in  problem  solving  is  to  encode  given  elements  of  problem-space 
(Newell  &  Simon,  1972).  Encoding  involves  identifying  most  salient  features  of  a 
problem  space,  storing  these  features  in  working  memory  and  retrieving  relevant 
infomiation  froin  long-term  memory.  A  problem-space  consists  of  the  external  world  as 
perceived,  internal  representations  of  these  perceptions,  and  resulting  interactions 
(Greeno,  1998).  From  a  situational  learning  perspective,  a  problein-space  has  intrinsic 
features,  or  affordances  as  advocated  by  Gibson  (1977).  For  Gibson,  affordances  entailed 
what  environment  offers  to  the  problem-solver,  both  good  (potentials)  and  bad 
(constraints)  qualities.  Thus,  affordances  are  not  invented  by  the  problem-solver; 
affordances  are  perceived  and  populate  objective  sense  of  situation.  Gibson's  perspective 
represents  an  extension  of  Dewey's  (1933/1998)  notion  that  a  problem  situation  provides 
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"unique  doubtfulness"  (p.  108),  that  is,  doubt  lies  not  in  the  mind,  but  in  the  situation. 
Further,  "The  nature  of  the  problem  fixes  the  end  of  thought,  and  the  end  controls  the 
process  of  thinking"  (p.  15). 

Problem  function.  An  ability  to  provide  a  robust,  high-fidelity  representation  of 
clinical  problem-space  is  instrumental  to  effective  problem-based  instruction  settings 
(Savery  &  Duffy,  1995).  In  particular,  the  extent  to  which  physical  environment  and 
social  context  are  salient  to  problem  situation  will  govern  student  perceptions  of  the 
authenticity  and  relevance  of  contextual  elements  (Banows,  1986).  The  selection  of 
problems  is  guided  by  curricular  objectives  (Barrows,  1996;  Boud  &  Feletti,  1997).  The 
student's  nced-to-know  is  stimulated  by  the  problem,  and  an  important  gauge  of  knowing 
is  the  ability  to  apply  knowledge  to  the  problem  (Norman,  1 988).  By  instructional 
design,  the  student  is  responsible  for  recognizing  the  need  to  know;  acquiring  the 
necessary  knowledge  and  skills,  and  applying  this  back  to  the  problem  in  a  reiterative 
process  (Barrows,  2000;  Savery  &  Duffy).  Students  identify  learning  issues  posed  by 
specific  problems  to  help  develop  understanding  about  underlying  concepts  and 
principles.  Thus,  knowledge  acquisition  and  understanding  are  constructed  via  working 
on  a  problem  ("concurrent  applicability")  rather  than  in  the  traditional  approach  in  which 
acquired  knowledge  is  a  prerequisite  to  working  on  a  problem  ("retrospective 
applicability"). 

From  this  vantage,  problem  interpretation  is  a  function  of  perception  and  learning 
to  perceive  situational  affordances  that  are  instrumental  to  developing  problem  solving 
expertise.  Dewey  (1933/1998)  and  Gibson  (1977)  further  imply  that  interpretation  does 
not  begin  where  perception  ends;  rather  efficacious  perception  adapts  and  emerges  with 
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action  in  the  problem-space.  Extending  these  positions,  Schon  (1987)  envisions  experts 
as  practitioners  who  efficaciously  link  perception,  interpretation  and  action  (e.g., 
knowing-in-action)  to  resolve  clinical  problems.  The  capacity  to  facilitate  an  optimal 
solution  or  to  improve  a  situation,  varies  with  the  problem-solver's  ability  to  inquire  and 
work  creatively  within  the  knowledge  and  skill  demands  of  a  problem  and  the 
affordances  (cues)  of  problem  space  (Grecno,  1998). 

Importantly,  a  health  care  problem  may  cue  an  enormous  range  of  associated 
knowledge,  much  of  which  will  be  irrelevant  to  achieving  a  solution  or  improvement. 
Cue  acquisition  is  the  process  of  collecting  data  in  clinical  problem  solving  (Elstein  et  al., 
1978).  For  medical  students  and  physicians,  research  suggests  accuracy  of  cue 
acquisition  and  interpretation  is  associated  with  accuracy  of  diagnosis  (Elstein  et  al.). 
Thus,  effective  cue  acquisition  and  interpretation  can  transform  an  ill-structured  problem 
into  a  well-structured  problem  (Simon,  1973).  Lovett  and  Anderson  (1996)  argue  that  the 
appropriate  use  of  problem-space  cues  and  eventual  problem  solving  success  are 
influenced  by  an  individual's  ability  to  integrate  processing  current  information  with 
processing  experienced-based  information,  histrumental  to  the  integration  of 
performance  and  learning  is  a  conceptual  cotipling  of  immediate  infoniiation  and 
successful  experience  in  a  manner  quantifiable  and  meaningful  (Lovett  &  Anderson). 
This  position,  which  reflects  the  essence  of  situated  learning  principles,  is  discussed  later. 

In  a  similar  vein,  Sternberg  (1994)  suggests  problem-solving  expertise  involves 
the  trajectory  through  memory  encoded  with  strong  context  specific  cues  to  limit  the 
retrieval  of  relevant  information  for  a  successful  solution  (i.e.,  encoding  specificity) 
(Lovett  &  Anderson,  1996).  Sternberg's  position  is  derived  from  evidence  that  indicates 
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successful  retrieval  is  contingent  upon  the  context  in  which  that  knowledge  was 
constructed  (Godden  &  Baddeley,  1975).  Context  dependence  is  demonstrated  when 
superior  recall  occurs  when  the  retrieval  situation  corresponds  to  the  situation  of  initial 
learning.  The  potency  of  context-dependent  learning  was  revealed  in  a  classic  study  by 
Godden  and  Baddeley  who  investigated  recall  in  Royal  Navy  divers.  One  group 
memorized  twenty  words  on  shore,  while  another  group  memorized  the  same  words 
while  submerged.  Words  memorized  on  shore  were  recalled  better  on  shore  and  words 
learned  submerged  were  better  recalled  underwater.  Thus,  memory  for  words  was 
significantly  better  in  the  context  of  original  learning.  Indeed,  encoding  specificity  as  an 
element  of  infomiation-processing  theory  (Lovett  &  Anderson)  resembles  context 
emphasis  in  a  situational  learning  perspective  (Albanese,  2000).  This  is  discussed  later. 
Learning  from  this  perspective  is  not  only  associated  with  the  acquisition  of  knowledge, 
but  also  its  representation.  Therefore,  as  indicated  earher,  efficiency  in  representing 
knowledge  for  re-deployment  is  viewed  as  a  primary  determinant  of  problem-solving 
expertise.  So,  the  flexible  application  of  knowledge  in  problem  solving  requires  that 
knowledge  elaboration  occur  transcontextually. 

Clearly,  knowledge  in  a  specific  context  is  necessary  for  initial  learning,  but  not 
sufficient  for  understanding.  The  implications  for  PBL  are  that  these  dimensions  must  be 
addressed  individually  to  create  a  learning  environment  conducive  to  knowledge 
construction,  relevant  retrieval,  and  appropriate  action  in  practice  (Schon,  1983,  1987). 
Guided  by  the  principle  of  prior  knowledge  activation,  advocated  by  both  information 
processing  theorists  (Anderson,  1990;  Ausubel,  1960)  and  social  constnictivists  (Savery 
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&  Duffy.  1995),  instruction  can  be  tailored  to  foster  linkage  (networking)  and  integration 

of  existing  knowledge  with  the  processing  of  new  infonnation. 

A  core  property  of  problem-based  teaching  and  learning  is  its  representational 

flexibility,  a  quality  that  facilitates  inferential  use  of  knowledge  in  novel  contexts 

(Barrows,  2000).  Theoretically,  perceiving  and  confronting  authentic  health  care 

problems  will  shape  a  medical  student's  clinical  reasoning  and  problem  solving  skills. 

When  students  are  challenged  with  such  a  complicated,  conceptual  process,  perception 

and  action  are  learned  together  as  a  unit  (Dewey,  1929a,  1948,  1933/1998). 

Extrapolating  this  insight  to  medical  education,  learner,  activity  and  environment 

mutually  constitute  the  other.  For  Dewey,  experience  is  the  initiating  phase  of  thought 

for  a  student,  as  students  need  an  empirical  situation  to  engage  their  interests  and 

generate  action.  Thus,  thought  is  intertwined  with  action.  Dewey  wrote  ".  .  .  old 

experience  is  used  to  suggest  aims  and  methods  for  developing  a  new  and  improved 

experience.  Consequently  experience  becomes  in  so  far  constructively  self-regulative" 

(1948,  p.  94).  Indeed,  Dewey's  stance  of  knowledge-in-use  is  grounded  in  his  conception 

of  educative  experience  and  education  as  growth  (Soltis,  1990).  Clarifying  Dewey's 

position,  Soltis  wrote: 

Previous  experience  that  enters  into  present  experience  to  inform  it,  organize  it, 
transform  it,  and  reconstruct  it  is  not  just  useful  knowledge  in  the  technical  sense 
of  knowing  how  to  do  something.  It  is  useful  in  the  richer  and  broader  sense  of 
being  able  to  use  one's  past  experience  to  orient  oneself  in  a  new  situation,  to 
interpret  its  manifold  dimensions,  to  analyze  its  components,  to  guess  at  or 
anticipate  its  future,  and  to  bring  one's  purpose  to  bear  on  the  ongoing  interaction 
of  self  and  situation,  (p.  311) 

Inherent  in  Dewey's  (1916,  1948)  educational  philosophy  is  the  activation  of 

relevant  prior  knowledge  as  a  prelude  to  constructing  ("intellectual  growth")  new 
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knowledge.  For  Schon  (1987),  Deweyan  inquiry  is  similar  to  the  concept  of  designing, 
"not  the  activities  of  the  'designing  professions'  such  as  architecture,  landscape 
architecture,  and  industrial  design,  but  the  more  inclusive  processes  of  making  things 
(including  the  representations  to  be  built)  under  conditions  of  complexity  and 
uncertainty"  (p.  30).  Examples  are  physician's  construction  of  a  diagnosis  or  a  teacher's 
construction  of  a  lesson  plan  (Schon). 

Also,  consistent  with  Dewey's  (1933/1998)  advice,  situated  learning  aims  to  teach 
knowledge  and  skills  employing  contextual  experiences  that  reflect  how  knowledge  will 
be  useful  in  practice  (Lave  &  Wenger,  1991).  If  development  ofa  student's  ability  to 
think  is  a  principal  goal  of  education,  then  students  must  learn  in  situations  where 
thinking  has  an  opportunity  to  occur  (Dewey).  Specifically,  Dewey  advocated  situations 
designed  to  foster  reflective  inquiry.  For  Dewey,  reflective  inquiry  mediated  experience. 
A  learning  environment  fostering  reflective  inquiry  would  include  a  genuine  situation  of 
experience  and  a  genuine  problem  developed  within  a  situation  as  a  stimulus  for  thinking. 
The  student  must  possess  cognitive  resources  to  make  observations  (perceive 
affordances),  to  work  the  problem  (problem-solving  strategies),  and  be  responsible  for 
suggesting  solutions  (propositions)  while  testing  those  propositions  to  reveal  their  value 
and  discover  their  validity  (Dewey).  Dewey's  conception  of  inquiry  resembles  scienfific 
method  and  integrates  theory  (abstract  representations)  and  practice  (Soltis,  1990).  That 
is,  reflective  inquiry,  oriented  vis-a-vis  established  theory,  develops  propositions  and 
tests  them  within  the  context  of  a  problem.  For  Dewey,  "To  grasp  a  meaning,  to 
understand,  to  identify  a  thing  in  a  situation  in  which  it  is  important,  are  thus  equivalent 
terms"  (p.  117).  Thus,  performance  in  some  situations  is  limited  by  a  tendency  to 
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recognize  problems  and  attune  to  affordances.  From  this  perspective  with  empirical 
support  (Godden  &  Baddeley,  1975),  the  power  of  knowledge  is  situation  specific  and 
influences  one's  mode  of  participation  in  a  problein-space.  Further,  situational 
empowerment  can  be  viewed  as  integrating  power  potential  (competence)  and  actual 
power  (performance)  (Bandura,  1986,  1997).  This  is  discussed  later. 

Influenced  by  Deweyan  tradition,  an  assumption  of  problem-based  instruction  is 
that  clinical  perception  is  an  active  rather  than  a  passive  process,  and  results  from 
attention,  prior  experience,  and  a  conscious  search  for  infomiation  (Barrows,  2000; 
Barrows  &  Tamblyn,  1980).  The  initial  perception  of  a  problem  situation  serves  as  a 
stimulus  for  solution  ideas,  thus  this  represents  a  crucial  phase  in  the  quality  of  clinical 
reasoning  (Barrows  &  Tamblyn).  A  challenge  for  medical  students  is  to  actively  attend 
to  and  interpret  situational  affordances.  That  is,  to  exhibit  situational  intelligence  and  be 
sensitive  to  evidence  (e.g.,  patient  signs  and  symptoms)  particularly  when  evaluating  and 
revising  solution  possibilities.  Such  a  process  of  thinking  considers  options  and  reasons 
(intelligent  search)  before  rendering  judgment  or  commencing  action  (intelligent 
hesitation),  because  some  impulsive  tendencies  or  reactions  "need  constant  regulation"  to 
obviate  premature  closure  (Dewey,  1933/1998,  p.  23).  Moreover,  such  a  disposition 
fosters  prospective  awareness,  which  can  be  viewed  as  Promethean  (wise  before  events). 
"To  be  genuinely  thoughtful,  we  must  be  willing  to  sustain  and  protract  the  state  of  doubt 
which  is  the  stimulus  through  inquiry,  so  as  not  to  accept  an  idea  or  make  positive 
assertion  of  a  belief  until  justifying  reasons  have  been  found"  (Dewey,  p.  16). 

According  to  the  principles  of  situated  learning,  instrtictional  methods  that 
employ  repeated  exposure  to  problems  increase  the  representational  richness  of  problems. 
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which  increases  the  learner's  abihty  to  identify  and  interpret  affordances,  and  understand 
problems  (Barrows,  1986,  2000;  Greeno,  1998;  Resnick,  1989,  1991)  as  they  transition 
from  concept  novice  to  concept  expert.  Iterations  are  important  because  students  will 
acquire  some  understanding  from  original  learning  situations,  but  will  miss  much  of  the 
detail  until  later  exposures  (Resnick).  Thus,  each  iteration  cycle  provides  new  insight, 
and  increases  opportunities  for  reflection  and  understanding  (meaningful  abstractions). 
For  Regan-Smith  and  associates  ( 1 994),  abstractions  allow  medical  knowledge  to  be 
remembered  and  retrieved.  Further,  abstraction  of  contextual  learning  is  thought  to  be 
important  vis-a-vis  knowledge  transfer  (Dewey,  1933/1998;  Regan-Smith  et  al;  Salomon 
&  Perkins,  1988). 

Salomon  and  Perkins  (1988)  have  characterized  transfer  as  "low-road  transfer" 
and  "high  road  transfer."  Low-road  transfer  involves  automated  (no  requirement  for 
reflection)  transfer  of  frequently  practiced  skills,  while  high-road  transfer  requires  the 
abstraction  of  knowledge  and  intentional  translation  to  accommodate  the  demands  of  a 
novel  situation.  According  to  Salomon  and  Perkins,  high-road  transfer  is  fostered  best 
when  instructional  conditions  encourage  students  to  engage  in  "mindful  abstractions,"  for 
example,  intentional  abstraction  of  a  principle  after  successfully  applying  the  principle  in 
varied  situations.  Thus,  students  acquire  knowledge  at  once  linked  to  situations  of 
deployment  and  stripped  of  context.  In  other  words,  learning  is  from  the  particular  to  the 
general  rather  than  vice  versa.  From  this  view,  learning  is  an  inductive  process  used  to 
construct  theories  of  use,  or  induction  into  practice  (Schon,  1987). 

Although  considered  a  pragmatist,  Dewey  (1933/1998)  viewed  abstract  thinking 
as  an  essential  element  of  a  complete  act  of  thought.   For  example  Dewey  wrote,  "When 
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thinking  is  used  as  a  means  to  some  end,  good  or  value  beyond  itself,  it  is  concrete;  when 
it  is  employed  simply  as  a  means  to  more  thinking,  it  is  abstract"  (p.  223).  Apparently, 
for  Dewey,  knowledge  constructed  by  reflective  inquiry  is  potentially  certain.  Thus, 
ideas  can  be  abstracted  from  particulars  of  experience  and  become  relatively  independent 
of  their  context  through  a  process  of  inductive  reasoning.  In  further  support  of  Dewey's 
position  advocating  purposeful  or  pragmatic  abstraction,  Salomon  and  Perkins  (1988) 
have  called  reflection  "high  road  to  transfer." 

Theoretical  support  of  this  position  is  emerging  from  what  previously  were 
diametrically  opposed  cognitive  (infonnation-processing  theory)  and  situated  learning 
perspectives  (Anderson.  Greeno,  Reder,  &  Simon,  2000;  Cobb  &  Bowers,  1999). 
Whereas  traditional  cognitive  theorists  focus  on  the  individual  as  a  unit  of  analysis, 
situated  learning  perspectives  focus  on  interactive  systems  that  incorporate  individuals  as 
participants,  interacting  with  each  other,  as  well  as  with  the  physical  characteristics  of  the 
learning  environment  or  problem-space  (e.g.,  materials  and  representational  systems) 
(Cobb  &  Bowers).  The  convergence  of  these  two  perspectives  appears  sine  qua  non  to 
move  beyond  the  dichotomy  in  medical  education  between  knowing  and  doing  and 
understanding  the  successful  integration  of  biomedical  science  and  clinical  practice, 
which  represents  the  utility  of  this  notion  of  dual  acquisition  of  knowledge.  Presumably, 
cognitive  flexibility  and  problem-solving  skills  transfer  will  be  facilitated  by  designing 
performance  challenges  that  require  similar  declarative  and  procedural  knowledge  in 
varied  problem-situations  (Anderson  et  al.;  Cobb,  2002;  Cobb  &  Bowers;  Perkins  & 
Salomon,  1989). 
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Problem  efficacy.  Problem  effectiveness  is  operatioiialized  as  the  degree  of 
congruence  between  student  generated  learning  issues  and  instructional  objectives 
(Mpofu,  Das,  Murdich,  &  Lanphear,  1997).  Research  outcomes  reveal  varying 
magnitudes  of  correspondence  between  instructional  objectives  and  student-generated 
learning  issues  prompted  by  posed  health  care  problem  (Dolmans,  Gijselaers,  Schmidt,  & 
van  der  Meer,  1993;  Mpofu  et  al.,  1997;  Wendelberger,  Simpson.  &  Biemat,  1996).  In  a 
randomized  study  of  second-year  medical  students  undergoing  a  six-week  course  in 
normal  pregnancy,  parturition,  and  child  development  in  a  PBL  curriculum,  Dolmans  and 
colleagues  observed  problem  effectiveness  varied  from  28%  to  100%,  with  an  average 
effectiveness  of  64%.  Interestingly,  6.2%  of  learning  issues  generated  were  unexpected, 
and  47%  of  those  issues  were  rated  by  faculty  to  be  either  moderately  relevant  or  relevant 
to  objectives.  Extending  this  line  of  research,  Wendelberger  et  al.  examined  the 
correlation  between  student  generated  learning  issues  and  PBL  cuuiculum  objectives  and 
observed  better  than  90%  agreement.  Similarly,  Mpofu  et  al.  observed  55  to  100%  (mean 
91.4%)  congruence  between  tutorial  group  generated  learning  issues  and  instructional 
objectives.  Quantifying  problem  effectiveness  affords  opportunities  to  identify  problems 
that  inadequately  stimulate  learning  issues. 

The  importance  of  problem  quality  was  quantified  further  by  van  Berkel  and 
Schmidt  (2000)  who  surveyed  1350  health  sciences  students  enrolled  in  120  courses  over 
six  years.  Data  were  collected  regarding  prior  knowledge  of  students,  quality  of 
problems  posed,  tutorial  group  effectiveness,  tutorial  group  attendance,  self-study  time, 
interest  in  subject  content,  and  academic  achievement.  Using  mtercorrelations  and  path 
analysis,  investigators  revealed  problem  quality  was  influenced  more  by  prior  knowledge 
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(.61 )  than  tutor  perfomiance  (.39).  The  effectiveness  of  a  tutorial  group  was  influenced 
more  by  problem  quality  (.45)  than  tutor  performance  (.37).  Tutorial  group  attendance 
was  effected  directly  by  tutorial-group  functioning  (.48)  and  inversely  by  amount  of  prior 
knowledge  (-.31)  or  quality  of  problems  (-.19).  Tutorial  group  attendance  was  greater  in 
students  who  needed  to  know  about  the  content  related  to  the  problein  or  who  perceived 
the  quality  of  the  problem  to  be  wanting.  Moreover,  an  inverse  relationship  between 
group  attendance  and  self-study  time  suggests  that  students  compensate  between  time 
allocated  to  collaboration  and  individual  study.  Intrinsic  interest  in  content  was 
influenced  more  by  problem  quality  (.68)  than  tutorial  group  effectiveness  (.32). 
Resultant  achievement  was  influenced  by  group  attendance  (.36),  quality  of  problem  (.32) 
and  self-study  time  (.14).  These  results  extend  and  confirm  previous  work  quantifying 
important  inter-relationships  and  the  multivariate  nature  of  problem-based  learning 
(Schmidt  &  Gijselaers,  1990;  van  Til,  van  der  Vleuten,  &  van  Berkel,  1997)  and  are 
discussed  later. 

Summary.  Congruent  with  the  stance  of  some  social  constructivists  (Savery  & 
Duffy,  1995)  and  situated  learning  theorists  (Brown  et  al.,  1989;  Greeno,  1998;  Lave  & 
Wenger,  1991;  Perkins,  1986),  interactions  between  students  and  multiple  features  of 
complex  learning  environment  created  by  problem-oriented  inquiry  co-influences  student 
activity,  motivation  and  achievement.  In  addition,  these  observations  align  well  with 
adult  learning  theory  (Knowles,  1975,  1984)  that  posits  adults  are  motivated  by  learning 
perceived  as  focused  on  relevant  problems  (e.g.,  face  validity),  participatory  with  active 
evolvement,  designed  so  that  students  have  responsibility  for  their  learning,  immediate 
applicability  to  practice  (e.g.,  concurrent  applicability),  and  based  on  cycles  of  action  and 
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reflection  (e.g.,  reiterative  or  recursive  cognitive  processing).  Problem-based  instruction 
capitalizes  on  these  values,  which  promote  leamer-centered  and  problem-oriented 
approaches  to  learning.  The  aim  is  to  develop  physicians  equipped  with  adult  learning 
skills  that  enable  them  to  adapt  to  varied  and  uncertain  practice  situations  (Dewey, 
1933/1998;  Schon,  1987). 
Role  of  Students 

"Curriculum  refomi,  if  it  is  to  have  any  chance  of  success,  inust  begin  with  the 
students.  .  .  "  (Nelms,  1993,  p.  94).  In  accordance  with  this  view,  students  undergoing 
problem-based  instruction  are  required  to  assume  responsibility  for  inanaging  their 
learning.  Unfortunately,  most  medical  students  have  spent  their  previous  academic  years 
assuming  the  teacher  was  the  principal  disseminator  of  knowledge  (Nomian  &  Schmidt, 
1992).  This  type  of  educational  process  can  be  characterized  as  one  in  which  students  are 
spoon-fed  (Dolmans,  Schmidt,  &  Gjiselears,  1995).  Research  suggests  this  lack  of 
independence,  reliance  on  the  expertise  of  the  instnictor,  and  rote  memorization  of  facts, 
may  cause  many  new  PBL  students  to  experience  significant  academic  stress  and  anxiety 
of  varied  etiology  (Bernstein  ct  al.,  1995;  Dyke,  Jamrozik,  &  Plant,  2001;  Gijselaers, 
1996;  Liebemian,  Stroup-Benham,  Peel,  &  Camp,  1997;  Moore,  Block,  Style,  & 
Mitchell,  1994;  Moore-West,  Harrington,  Mennin,  Kaufman,  &  Skipper,  1989;  Moore- 
West  &  O'Donnell,  1985;  Nonnan  &  Schmidt,  1992). 

Cognitive  dissonance.  Research  results  are  mixed  regarding  how  students  cope 
with  cognitive  conflict  in  problem-based  learning  environments.  Moore-West  and 
O'Donnell  (1985)  conducted  a  quasi-experimental,  longitudinal  study  comparing  learning 
environment  anxiety  in  students  undergoing  PBL  versus  LBL.  They  employed  a 
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questionnaire  witli  established  validity  and  reliability  to  measure  different  dimensions  of 
anxiety  at  entrance  to  medical  school  and  again  two-years  later.  Despite  equivalent 
anxiety  scores  at  entrance  to  medical  school  and  controlling  for  Medical  College 
Admission  Test  scores,  gender  and  ethnicity,  mean  anxiety  level  changed  little  in  PBL 
students  (M  =  16.41,  SD  =  8.41),  but  the  level  was  significantly  greater  in  LBL  students 
(M  =  27.57,  SD  =1 5.61,  t  =  4.32,  e  <  .0001 ).  In  a  follow-up  study  of  similar  design, 
Moore-West  et  al.  (1989)  confirmed  and  extended  earlier  results  and  concluded  a 
curriculum  incorporating  PBL  versus  LBL  manifests  less  academic  stress  associated  with 
mastering  an  enormous  knowledge  and  skills  base  while  more  effectively  preparing 
students  to  manage  stress  situations  similar  to  those  encountered  by  practicing  physicians. 

Additional  support  for  the  differential  effect  of  PBL  and  LBL  on  student 
perception  of  academic  stress  is  provided  by  Lieberman  et  al.  (1997).  They  conducted  a 
quasi-experimental  study  (pre-post  test  design)  employing  valid  and  reliable  equivalent 
forms  of  the  Medical  School  Learning  Environment  Survey  (Feletti  &  Clark  1981a, 
1981b)  and  revealed  significantly  less  stress  and  significantly  more  positive  attitude 
toward  learning  in  similarly  matched  first-year  PBL  versus  LBL  students  undergoing 
simultaneous  parallel  curricula  in  the  same  institution. 

Academically  more  mature  medical  students  were  studied  by  Bernstein  and 
colleagues  (1995)  who  reported  second  year  students  undergoing  transition  from  LBL  to 
PBL  initially  expressed  concern  that  PBL  would  result  in  "knowledge  gaps,  reinforce  the 
wrong  information,  and  make  inefficient  use  of  valuable  time"  (p.  245).  In  fact,  after  the 
first  PBL  sessions  (5-weeks),  these  students  exhibited  a  shift  in  perceptions  from  38%  to 
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52%  agreeing  PBL  was  more  relevant  and  effective  than  LBL  (chi  squared  =  6.5,  g.< 
.02). 

Further  insights  were  provided  by  Moore  et  al.  (1994)  who  used  the  Cognitive 
Behavior  Survey  (Mitchell,  1994)  to  obtain  perceptions  in  second  year  students  who  had 
been  assigned  randomly  to  either  PBL  or  LBL  at  Harvard  Medical  School.  The 
Cognitive  Behavior  Survey  assesses  medical  student  learning  in  preclinical  years 
incorporating  1 15  Likert-scale,  open-ended,  fill-in,  and  rank-order  items  with  subscales 
including  learning  behavior,  nature  of  learning,  and  epistemological  stance.  Chronbach's 
alpha  for  the  subscales  was  .82,  .79,  and  .76,  respectively.  Problem-based  learning 
students  particularly  were  anxious  regarding  "what  and  how  much  to  study"  (p.  988), 
which  adversely  effected  the  dynamics  of  small  group  tutorial  sessions,  a  key  component 
of  the  PBL  process.  Approaching  graduation,  these  students  again  were  surveyed  and 
asked  to  select  key  words  that  described  their  preclinical  experience.  "Stressful, 
engaging,  and  difficult"  and  "nonrelevant,  passive,  and  boring"  were  descriptors  cited  by 
PBL  and  LBL  students,  respectively,  statistically  more  often  (g_<  .05).  Although 
instructionally  stressful  for  students,  authors  believed  PBL  experiences  presented 
challenges  comparable  to  those  faced  in  clinical  practice,  thus  contributing  to  the 
authenticity  of  the  learning  situation.  Moreover,  tolerance  of  ambiguity  was  significantly 
greater  in  PBL  versus  LBL  students  (e  =  04).  hi  addition,  students  undergoing  PBL 
scored  higher  on  a  subscale  assessing  reflection  (p  <  .001 )  and  lower  on  a  subscale 
assessing  memorization  (g  =  .01). 

Dyke  and  associates  (2001 )  in  a  randomized  trial  found  PBL  students  initially 
expressed  anxiety  regarding  self-directed  learning  and  engaging  in  responsible  learning 
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activities  witli  peers  (detailed  results  and  analysis  presented  in  Chapter  4).  However,  as 
students  gained  experience  with  PBL,  anxiety  waned,  and  on  average,  nearly  vanished  by 
the  end  of  the  course.  Moreover,  PBL  students'  positive  perception  oftheir  tutorial- 
group's  commitment  to  learning  increased  progressively  through  the  course.  Students, 
however,  frequently  noted  that  problem-based  instruction  was  a  very  time-consuming 
style  of  learning. 

Active  learning.  For  cognitive  constructivists,  "cognitive  disequilibrium" 
(Piaget,  1969)  or  "cognitive  dissonance"  (Savery  &  Duffy,  1995)  is  required  to  motivate 
self-orchestrated,  active  learning.  Coherent  with  these  beliefs,  problem-based  instruction 
is  designed  to  create  what  Dewey  (1933/1998)  called  a  "problematic  situation"  (p.  102) 
and  invites  students  to  engage  problems  in  a  process  mirroring  reflective  inquiry  (Savery 
&  Duffy).  For  Dewey,  reflective  inquiry  is  a  scientific  method  in  which  problems  are 
identified,  problems  are  studied  via  active  engagement,  and  conclusions  (warranted 
assertions)  are  reached,  as  problems  are  resolved  tentatively.  That  is,  problem  solutions 
are  tentative  until  better  information  becomes  available.  Dewey  (1929b)  was  committed 
to  employing  science  as  a  method  for  solving  problems  and  learning. 

Command  of  scientific  methods  and  systemized  subject  matter  liberates 
individuals;  it  enables  them  to  see  new  problems,  devise  new  procedures,  and,  in 
general,  makes  for  diversification  rather  than  for  set  uniformity.  But  at  the  same 
time  these  diversifications  have  a  cumulative  effect  in  an  advance  shared  by  all 
workers  in  the  field,  (pp.  12-13) 

In  addition,  reficctive  inquiry  allows  practitioners  to  know  possible  problems  and 

their  solutions  in  advance  to  anticipate  and  influence  their  outcomes.  Dewey 

(1933/1998)  described  this  habit  as  "pre-reflective"  (p.  106).  Prospective  awareness,  or  a 

Promethean  disposition  is  more  likely  to  cultivate  a  practitioner  who  interprets  (transacts) 
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situations  rather  tlian  reacts  to  situations  (Dewey).  Instruction  embracing  this  philosophy 
would  be  designed  to  facilitate  methods  of  cognition,  not  didactics,  because  methods  of 
cognition  can  be  employed  to  improve  situations  involving  uncertain  outcomes,  limited 
guidance,  limited  contextual  knowledge,  and  the  need  to  consider  multiple  perspectives 
(Dewey,  1916;  Spiro,  Feltovitch,  Jacobson,  &  Coulson,  1992).  Indeed,  according  to  Karl 
Pearson,  "The  true  aim  of  the  teacher  should  be  to  impart  an  appreciation  of  method 
rather  than  a  knowledge  of  facts"  (cited  in  Pickering,  1956,  p.  116).  Such  a  pedagogy 
strongly  parallels  scientific  method  and  reflective  inquiry  (Dewey,  1933/1998)  and 
should  increase  situational  competence,  particularly  in  indeterminate  clinical  problem- 
spaces  (Schon,  1987),  which  is  a  feature  of  PBL  (Barrows,  1986).  Dewey  (1916)  may 
have  described  this  method  best: 

What  is  meant  by  calling  a  method  intellectual?  Take  the  case  of  a  physician.  No 
mode  of  behavior  more  imperiously  demands  knowledge  of  established  modes  of 
diagnosis  and  treatment  than  does  his.  But  after  all,  cases  are  like,  not  identical. 
To  be  used  intelligently,  existing  practices,  however  authorized  they  may  be,  have 
to  be  adapted  to  the  exigencies  of  particular  cases.  Accordingly,  recognized 
procedures  indicate  to  the  physician  what  inquiries  to  set  on  foot  for  himself,  what 
measures  to  try.  They  are  standpoints  from  which  to  carry  on  investigations;  they 
economize  a  survey  of  the  features  of  the  particular  case  by  suggesting  the  things 
to  be  especially  looked  into.  The  physician's  own  personal  attitudes,  his  own 
ways  (individual  methods)  of  dealing  with  the  situation  in  which  he  is  concerned, 
are  not  subordinated  to  the  general  principles  of  procedure,  but  are  facilitated  and 
directed  by  the  latter,  (p.  201-201) 

Reflective  inquiry  appears  to  be  a  suitable  methodological  framework  to  guide  a 

student's  role  (operational  conduct)  in  a  problem-based  learning  environment.  Reflective 

inquiry  is  a  strategy  for  knowing  that  incorporates  a  process  for  analyzing,  comparing, 

and  contrasting  concepts  and  their  implications  for  action,  or  practice.  Reficctive  inquiry 

is  both  a  means  for  adapting  and  a  tool  to  generate  "warranted  assertions."  Thus, 

Dewey's  theory  of  inquiry  involves  thought  intertwined  with  action  to  complete  an  act  of 
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thought.  That  is,  reflective  experience,  which  continues  from  doubt  and 
indetemiinateness  to  resolution  of  doubt,  to  generation  of  new  doubt  (Dewey,  1948. 
1933/1998).  This  method  of  thinking  involves  a  reiterative  process  of  clarification, 
testing,  and  refinement. 

For  Dewey  (1933/1998),  uncertainty  is  not  confined  to  the  mind,  but  involves 
situation  because,  if  doubt  were  only  in  the  mind,  cognitive  processes  alone  would 
resolve  uncertainty.  Reflective  inquiry  commences  with  situations  that  are  problematic 
and  obstruct  meaningful  action.  The  inquirer  frames  the  problem  and  formulates  a 
strategy  of  action.  Consequently,  the  inquirer  is  immersed  in,  and  in  transaction  with,  a 
problematic  situation,  which  is  a  perspective  held  by  situational  learning  theorists  (Brown 
et  al.,  1989;Greeno,  1998).  Personal  actions  change  elements  of  the  situation.  That  is, 
actions  can  open  or  close  opportunities,  rendering  subsequent  action  more  or  less 
plausible,  respectfully.  Therefore,  the  consequences  of  action  can  be  either  resources  or 
obstacles  in  the  acted  in  and  upon  situation  (Schon,  1983).  Consequence  (end)  of  action 
(means)  becomes  the  resource  (means)  or  an  obstacle  in  changed  situation.  Quality  or 
satisfactoriness  of  "end"  deteniiines  its  potential  as  a  subsequent  "means"  in  facilitating  a 
repertoire  of  new  possibilities.  For  Dewey,  an  "end"  not  envisioned  as  a  "means"  in 
facilitating  new  possibilities  was  ostensibly  a  cul-de-sac,  or  obviated  a  complete  act  of 
thought. 

Reflective  inquiry  is  not  just  a  useful  tool,  but  also  an  organizing  principle  for 
processing  infomiation  and  constructing  and  re-constructing  knowledge  (i.e., 
operationalized  intelligence).  As  indicated  earlier,  Salomon  and  Perkins  (1988)  claim 
that  reflection  is  essential  for  high-road  to  transfer.  Indeed,  the  evidence  suggests 
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successful  learners  and  problem-solvers  employ  "deep  processing"  (Coles,  1985,  1990, 
1997;  Craik  &  Lockhart,  1972;  Newble  &  Entwistle,  1986;  van  Til  et  al.,  1997),  a  method 
approximating  reflective  inquiry.  When  students  employ  deep  cognitive  processing  to 
attain  understanding,  their  behavior  is  characterized  by  the  logical,  systematic 
examination  of  evidence  and  interaction  of  evidence  (Newble  &  Entwistle).  Deep 
processing  involves  critical  thinking  rather  than  immediate,  uncritical  assent  to  certainty, 
which  forces  premature  closure  and  entrance  into  an  intellectual  cul-de-sac  (Dewey, 
1933/1998).  Students  employing  deep  cognitive  processing  scrutinize  contradictory 
infomiation,  attempting  to  understand  plausible  alternative  explanations,  elaborating 
interrelationships  between  evidence,  competing  explanations,  and  considering  the  fullest 
range  of  evidence  (Newble  &  Entwistle). 

Student  activity,  van  Til  and  others  (1997)  further  clarified  the  roles  of  students 
engaged  in  problem-based  learning.  They  distinguished  two  dimensions  of  student 
behavior  characterized  as  (a)  a  function  of  learning  style,  using  the  levels  of  processing 
framework  (Craik  &  Lockhart,  1972),  and  (b)  activity  in  students,  as  active  or  passive 
participants  during  tutorial-group  meetings.  Combining  the  anchors  of  these  two 
dimensions-activity  of  PBL  behavior  and  style  of  PBL  behavior  (surface  cognitive 
processing  or  deep  cognitive  processing)-four  behavior  combinations  were  generated 
including  deep-active,  surface-active,  deep-passive,  and  surface-passive.  The  authors 
observed  in  second  (n  =  46),  third  (n  =  30),  and  fourth  (n  =  57)  year  medical  students 
differential  achievement  on  a  written  progress  test  as  a  function  of  activity  and  style 
dimensions  of  PBL  behavior.  Inter-group  scores  regressed  on  year  of  training  dimension 
were  statistically  similar,  thus  the  data  were  pooled  for  summary.  Highest  achievement 
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scores  were  obtained  by  students  exhibiting  deep-active  behavior  (M  =  62.6,  SD  =  18.0) 

and  lowest  scores  were  observed  in  students  with  surface-passive  behavior  (M  =52. 1 ,  SD 

=  15.5)  (Effect  size  =  0.63).  However,  regression  analysis  of  PBL  behavior  dimensions 

revealed  that  only  the  activity  dimension  was  significantly  associated  with  achievement 

(R"  =  .28,  g  <  .05).  The  differential  effect  of  each  dimension  was  most  pronounced  in 

second-year  and  least  in  fourth-year  students.  These  results  imply  that  student 

achievement  is  influenced  significantly  by  activity  patterns  during  PBL.  Moreover, 

fomiatively  tracking  these  dimensions  may  inform  educators  about  instructional  effects 

on  the  learning  process.  Implications  of  this  research  are  important  because  they  support 

the  argument  that  learning  outcomes  research  is  functionally  meaningless  unless 

associated  strategies  for  learning  are  revealed  (Dubin  &  Taveggia,  1968). 

Interestingly,  the  methodology  and  results  reported  by  van  Til  et  al.  ( 1 997) 

illustrate  that  investigating  instrtictional  intervention  solely  with  achievement  outcomes 

unmasks  only  part  of  the  learning  process  (Dubin  &  Taveggia,  1968).  Further,  ten  Gate 

(2001)  suggests: 

In  educational  outcome  research,  a  research  design  with  an  educational  method  as 
an  independent  variable  and  a  curricular  test  as  the  independent  variable,  however 
elegantly  it  is  conducted  and  however  pristine  its  methodological  aspects,  is 
usually  too  simple.  Actually,  the  test  operates  as  a  dependent  variable  and 
independent  variable  at  the  same  time;  the  test  is  part  of  the  curriculum,  (pp.  85- 
86) 

These  results  complement  research  indicating  that  the  principal  cause  of  learning  is 
student  activity,  and  not  instructional  approach  (Dubin  &  Taveggia;  ten  Gate;  van  Til  et 
al.). 

Active  behavior  and  deep  cognitive  processing  (reflection)  appear  appropriate 
strategies  to  enhance  a  student's  ability  to  transact  efficiently  and  successfully  their  role 
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in  problem-based  teaching  and  learning.  Moreover,  deep  processing  also  is  an 
operational  feature  of  metacognition  (Barrows,  2000).  Flavel  (1987)  distinguished  three 
elements  of  mctacognitive  knowledge  including  knowledge  about  self,  knowledge  about 
task,  and  strategic  knowledge.  Thus,  reflection  appears  a  fundamental  feature  of 
metacognition,  and  a  necessary  condition  for  achieving  intentional  regulation  of  learning. 
As  previously  suggested,  reflective  inquiry  (Dewey,  1933/1998)  is  a  method  of 
metacognition,  that  is,  thinking  abotit  thinking.  Thus,  such  a  disposition  seems  tailored 
for  adoption  by  students  to  facilitate  the  development  of  strategies  for  effectively 
executing  their  role  in  PBL.  Moreover,  from  a  cognitive  and  instructional  theory 
perspective,  an  effective  inquiry  method  and  comprehension  monitoring  are  desired 
learning  processes  (Savery  &  Duffy,  1995)  and  play  an  important  role  for  medical 
students  in  coming  to  understand  professional  practice  (Schon,  1987). 

Dewey  considered  reflective  inquiry  instrumental  to  obviating  dichotomy  between 
learning  and  doing  and  providing  a  method  to  enable  intelligent  action.  "The  value  of 
any  cognitive  conclusion  depends  upon  the  method  by  which  it  is  reached,  so  the 
perfecting  of  method,  the  perfecting  of  intelligence,  is  the  thing  of  supreme  value" 
(Dewey,  1929a,  p.  200).  An  aim  of  medical  education  is  to  enable  medical  students  and 
physicians  to  analyze  infomialion,  to  improve  their  problem-solving  skills,  and  to  reflect 
their  learning  process  (Liaison  Committee  on  Medical  Education,  2002). 

Summary.  While  self-directed  and  collaborative  learning  responsibilities  can 
precipitate  an.\iety,  principles  of  cognitive  psychology  suggest  anxiety  can  promote 
greater  effort  and  enhance  performance,  both  collectively  and  individually  (Bandura, 
1986,  1997).  Moreover,  if  adopted,  Dewey's  method  of  reflective  inquiry  can  equip 
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students  with  a  psychological  tool  to  search  efficiently  and  effectively  for  goodness-of-fit 

between  problem-representation  and  contextual  data.  For  Fedemian  (1999): 

Implicit  in  PBL  and  the  tutorial  process  is  an  awesome  respect  for  the  beginning 
student.  The  recognition  that  one  is  contributing  to  the  learning  of  the  whole 
group  can  be  by  turns  inspiring,  threatening,  and  overwhelming.  .  .  .  Given  the 
underlying  insecurity  of  most  medical  students,  this  regard  for  their  potential  is 
enormously  positive,  (p.  93) 

Knowing  how  to  manage  the  paradoxical  tension  between  inducing  cognitive  dissonance 

while  providing  cognitive  scaffolding  is  essential  for  efficacious  tutoring. 

Role  of  Small  Group  Tutorial 

Rogoff  (1991)  describes  a  collaborative  model  of  learning  as  "guided 
participation"  where  students  are  active  participants  with  guidance  from  more 
knowledgeable  others.  Through  mutual  engagement,  learners  and  their  peers  manage 
activity  with  guidance  from  a  tutor.  This  form  of  participation  shares  many 
characteristics  of  cognitive  apprenticeships  described  by  Brown  and  colleagues  (1989) 
including  communication  (modeling),  bridging  student's  current  understanding  and  skills 
with  new  ones  (coaching),  where  problem  solving  is  structured  so  that  students  are 
supported  and  challenged  in  the  tasks  they  undertake  (through  the  interaction).  Key  to 
this  view  of  guided  participation  is  the  titrated  transfer  of  responsibility  as  skills  are 
developed  and  deployed  by  students.  A  salient  distinction  in  the  structure  of  lecture- 
based  versus  problem-based  instmction  is  the  amount  of  time  scheduled  for  formal, 
group-based  learning  (Richards  &  Cariaga,  1993). 

Principles  of  constructivism  and  situated  learning  predict  collaborative  discussion 
will  facilitate  the  elaboration  of  prior  knowledge  and  promote  construction  of  richer  more 
meaningful  knowledge  representations  of  problem  challenges  and  associated  information. 
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In  particular,  collaborative  learning  assumes  knowledge  is  situational  and  socially 
negotiated  (Vygotsky,  1926/1997),  rather  than  individually  constnicted,  by  communities 
of  learners  (Brown  &  Canipione,  1990)  and  that  generation  and  testing  of  information 
and  ideas  are  a  process  in  which  anyone  can  participate  (Brown  et  al.,  1989;  Lave  & 
Wenger,  1991 ).  Collaboration  distributes  the  cognitive  load  among  group  members  and 
permits  group  en  loto  (collective  knowledge)  to  engage  problems  that  demand  knowledge 
and  skills  (e.g.,  learning  issues)  exceeding  the  immediate  ability  of  individuals  (Brown  & 
Campione;  Gardner  &  Hatch,  1989;  Stage  et  al,  1998).  Informed  by  principles  of 
differential  psychology  (Knuth  &  Cunningham,  1993;  Ryan,  1999;  Sclion,  1987),  the 
concept  of  distributed  cognition  recognizes  not  every  member  of  a  tutorial  group  will  be 
ready  to  learn  material  simultaneously  (Brown,  1994;  Brown  &  Campione). 
Appreciating  the  varied  intellectual  standpoints  and  methods  governing  the  dynamics  of 
students  in  a  learning  environment  has  important  implications  for  medical  teaching  and 
learning. 

Capitalizing  on  personal  differences  and  perspectives,  collaboration  enlists  the 
collective  expertise  of  the  group,  as  learning  issues  are  distributed  among  members,  who 
through  self-directed  learning  become  quasi-experts  in  their  assigned  learning  issues 
(Brown,  1994).  Encouraging  students  to  become  experts,  or  "major"  in  varied  aspects  of 
content  provides  flexibility  with  respect  to  individual  learner  development  (Brown,  p.  7). 
Collaboration  also  fosters  idea  sharing  and  encourages  individuals  to  consider  and 
coordinate  varied  points  of  view,  thereby  facilitating  reasoning  and  higher-order  thinking 
(e.g.,  analysis,  syntheses,  and  evaluation)  and  promoting  elaborated  knowledge 
construction  (Bloom,  1984;  Resnick,  1989,  1991;  Ryan,  1999;  Schmidt,  1983). 
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Ryan  (1999)  posits  the  cognitive  constructivists'  Principle  of  Multiplicity  as  a  key 
framework  for  understanding  collaborative  knowledge  construction  during  problem- 
based  instruction.  For  Ryan,  the  Principle  of  Multiplicity  accentuates  the  importance  of 
dialogue  among  tutorial  group  members.  Different  from  argumentation,  dialogue  is  a 
process  used  to  facilitate  clarity  and  understanding.  Dialogue  is  not  simply  a 
communication  technique,  but  a  principle  grounded  in  the  premise  that  problem 
perception,  analysis,  and  resolution  is  coupled  inextricably  with  a  core  of  common 
understanding  (Dewey,  1933/1998;  Schon,  1987). 

Insight  for  dialogue  is  provided  by  Dewey  ( 1 933/1 998)  who,  as  discussed 
previously,  advocated  reflective  inquiry  as  a  method  for  enabling  students  to  suspend 
their  assumptions  and  the  process  of  responding  reflexively  or  habitually.  Thus,  students 
have  to  know  how  and  when  to  wake  up  their  patience  and  be  sensitive  to  relevant 
evidence.  This  requires  practice.  For  Dewey,  reflective  inquiry  is  a  mode  of  conduct  for 
collaborative  problem  solving.  Initial  response  is  arrested  (intelligent  hesitation)  to 
assess  and  clarify  the  problem  confronting  the  group.  For  collaborators  to  achieve 
correspondence,  varied  points  of  view  must  be  amalgamated  (Lave  &  Wenger,  1991). 
This  "attitude  represents  what  Mr.  Pierce  has  happily  tenned  the  'laboratory  habit  of 
mind'"  (Cited  in  Dewey,  1916,  p. 306). 

Group  activity  provides  students  with  the  opportunity  for  self-renection  and  joint 
construction  of  knowledge  (Brown  &  Campione,  1990;  Brown  &  Palinscar,  1989;  Lave 
&  Wenger,  1991).  Theoretically,  individual  reflective  inquiry  can  enhance  the  power  of 
the  group  to  achieve  the  desired  results.  Thus,  knowledge  activation  and  elaboration, 
dialogue  with  other  students,  and  refining  knowledge  gennane  to  the  problem  at  hand,  are 
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not  only  preparatory  steps  for  individual  learning,  they  also  are  instrumental  in  a 
cooperative  type  of  learning  that  fosters  restructuring  of  knowledge  or  conceptual  change 
(Bransford,  Brown,  &  Cocking,  1999;  Brown,  1994;  Brown  et  al.,  1989;  Schon,  1987). 
Moreover,  this  democratic  model  contrasts  starkly  with  traditional  teacher-centered 
learning  because  it  prescribes  a  shared  process  of  educational  dialogue,  negotiation, 
compromise,  and  consensus  (Dewey,  1916). 

Evidence  for  collaboration-induced  elaboration  in  college  science  students  is 
provided  by  Schmidt  et  al.  (1989).  Randomly  assigned  students  in  small  groups  were 
presented  with  either  a  problem  related  to  or  not  related  to  a  science  text  subsequently  to 
be  studied.  For  a  specified  tiine,  students  discussed  the  problem,  then  studied  the  text 
that  provided  novel  information.  Students  participating  in  relevant  problem  discussion 
before  studying  the  science  text  exhibited  significantly  better  free-recall,  explanations 
and  descriptions  (M  =  27.2,  SD  =  9. 1 ,  n  =  20  versus  M  =  1 9.7,  SD  =  7.5,  n  =  1 9,  p  =  .008, 
Effect  size  =  0.90).  Thus,  prospective  group  discussion  of  a  problem  matched  with 
incomplete  prior  knowledge  appears  an  appropriate  enabling  condition  for  fostering 
subsequent  knowledge  construction. 

The  achievement  effect  of  collaborative  learning  also  has  been  studied  using 
medical  students.  M^rtenson,  Eriksson,  and  Ingelman-Sundberg  (1985)  designed  a 
medical  chemistry  course  to  sustain  traditional  content  but  using  a  modified  instructional 
method  to  induce  students  to  obtain  primary  facts  themselves,  to  stimulate  problem 
solving  in  small  group  settings,  to  foster  understanding  by  asking  students  to  integrate 
chemistry  and  clinical  practice,  and  to  foster  responsibility  for  self-directed  learning.  The 
results  of  their  instructional  intervention  were  compared  to  a  historical  control,  that  is. 


72 

achievement  exhibited  by  students  completing  the  course  in  lecture-based  format  with 
similarly  structured  assessments.  Mean  essay  questions  achievement  was  72.7%  and 
60.7%  in  students  undergoing  problem-oriented,  collaborative  (1 1 -cohorts,  n  =  1651) 
versus  traditional,  lecture-based  learning  (5-cohorts,  n  =  818),  respectively.  This  is  an 
effect  size  of  0.26  with  a  95%  confidence  interval  of  0.18  to  0,34.  Mean  short  answer 
questions  achievement  was  72.2%  and  76.7%  in  students  undergoing  problem-oriented, 
collaborative  (1 1 -cohorts,  n  =  1651)  versus  traditional,  lecture-based  learning  (5-cohorts, 
n  =  818),  respectively.  This  is  an  effect  size  of -0.10  with  a  95%  confidence  interval  of- 
0.18  to  -0.02.  Mean  final  score  for  the  course  was  71.8%  and  68.2%  in  students 
undergoing  problem-oriented,  collaborative  versus  traditional,  lecture-based  learning, 
respectively.  This  is  an  effect  size  of  0.08  with  a  95%  confidence  interval  of  0.00  to  0.16. 
Results  are  limited  by  validity  threats  including  unsubstantiated  baseline  equivalence  and 
unknown  temporal  effects  associated  with  using  a  historical  control.   However, 
significantly  better  essay  question  scores  for  students  undergoing  collaborative  group- 
based  versus  lecture-based  learning  is  consistent  with  some  views  of  situated  learning 
(Brown  et  al.,  1989;  Savery  &  Duffy,  1995)  suggesting  active,  socially  mediated  learning 
facilitates  deep  processing  and  concept  elaboration. 

Further  evidence  is  provided  by  Lawry,  Schuldt,  Kreiter,  Densen,  and  Albanese 
(1999)  who  conducted  a  randomized  investigation  in  second-year  medical  students  and 
physician  assistants.  Instruction  in  methods  to  perfomi  a  musculoskeletal  examination 
(physical  examination)  was  completed  in  four  formats  including  written  materials  only, 
written  materials  and  videotape,  written  materials  and  small-group  sessions  facilitated  by 
senior  medical  students,  and  all  three  methods.  Facilitators  were  trained  by  one  member 
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of  the  faculty.  Written  materials  described  musculoskeletal  problems  (e.g.,  deformity, 
muscle  atrophy)  and  the  video  presented  examples  of  the  problems  and  corresponding 
physical  examination  procedures.  Achievement  was  quantified  using  a  written  test  and 
objective  standardized  physical  examinations  before  instructional  sessions,  and  then 
seven  to  ten  days,  three  months,  and  16  months  after  instruction.  Instructional  validity 
was  confirmed  by  an  improvement  of  2.5  and  5.0  standard  deviations  in  mean  written  and 
standardized  patient  examinations,  respectively,  between  the  mean  pre-course  scores  and 
all  students  independent  of  instructional  group  at  seven  to  ten  days.   Intergroup  written 
examination  achievement  was  statistically  similar.  However,  students  undergoing  small- 
group  instruction  exhibited  significantly  better  patient  examination  skills  compared  to 
those  who  underwent  written  material  or  videotape  instruction  at  each  of  the  examination 
periods.  The  addition  of  small-group  sessions  to  written  materials  affected  standardized 
patient  examination  scores  1.8  standard  deviations  greater  than  the  mean  score  observed 
for  written  materials  alone  and  1 .0  standard  deviation  above  value  achieved  for  written 
material  and  videotape  instruction.  The  authors  concluded  small-group  instruction  with 
hands-on  supervised  practice  is  superior  to  passive  instruction  for  teaching 
musculoskeletal  examination  skills,  and  this  superior  advantage  is  sustained  for  at  least 
sixteen  months.  Taken  together,  these  findings  and  those  of  Martenson  and  associates 
(1985),  illustrate  how  varied  achievement  measures  can  provide  varied  interpretation  of 
instructional  interventions;  this  is  discussed  later. 

Boehler  ct  al.  (2001)  designed  a  study  to  test  the  hypothesis  that  students  utilizing 
collaborative  study  methods  exhibit  greater  achievement.  Third-year  medical  students 
were  surveyed  to  quantify  reading  habits,  utilization  of  lectures,  group  study,  study 


74 

patterns,  and  resources  employed  during  five  consecutive  ten  week  surgery  clerlcships. 
Assessments  during  the  clerl<sliip  included  the  surgery  subtest  of  NBME  (NBME-II-S) 
and  the  muhiple  stations  clinical  examination  (MSCE).  The  MSCE  is  a  nineteen  station 
assessment  employing  standardized  patients  and  interpretation  of  clinical  information 
situated  in  the  framework  of  clinical  cases  emphasizing  problem  solving.  Group  study 
was  selected  by  18  (22%)  and  independent  study  by  63  (78%)  of  the  participants.  Groups 
comprised  two  or  three  students  and  convened  two  to  five  (55.5%),  six  to  ten  (16.7%),  or 
1 1  to  22  (27.8%)  times  per  month.  Students  from  first  and  second  year  PBL  track 
comprised  33.3%  of  participants.  Surprisingly,  of  twenty-seven  students  who  underwent 
reiterative,  tutorial-group  PBL,  only  eight  (30%)  chose  collaborative  learning.  Of  fifty- 
four  previous  LBL  students,  forty-four  selected  individual  study  (81%)  and  ten  (19%) 
selected  collaborative  learning.  Indeed,  there  was  no  significant  relationship  between 
whether  students  underwent  pre-clerkship  PBL  or  LBL  and  their  study  patterns  and  use 
of  resources,  except  PBL  students  reported  more  frequent  Internet  access  (p  =  .05). 
However,  investigation  revealed  superior  MSCE  achievement  for  students  who  studied  in 
groups  (p  =  .001 ).  Collaborative  learning  afforded  no  advantage  for  performance  on 
NBME-II-S.  Since  a  primary  goal  of  PBL  is  to  foster  appreciation  for  collaboration  in 
practice,  educational  implications  of  this  research,  if  corroborated,  are  significant. 
Possibly  more  important  for  medical  education  in  general,  results  of  this  research  suggest 
collaborative  learning  enhances  clinical  reasoning  competence,  independent  of  previous 
collaborative  learning  experience. 

De  Grave,  Boshuizen,  and  Schmidt  (1996)  studied  cognitive  and  metacognitive 
processes  during  problem  analysis  by  examining  conversations  (verbal  protocols) 
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between  group  members  and  their  thinking  processes.  Thinking  processes  were  revealed 

through  stimulated  recall  methods  (recall  protocols).  Analysis  of  verbal  protocols 

revealed  that  considerable  time  was  expended  for  data  exploration  and  problem 

formulation  coupled  with  a  dominant  tendency  to  construct  theories,  generate  causal 

relationships,  and  test  hypotheses.  Investigators  concluded  students  in  a  problem-based 

collaborative  process  undergo  alternating  periods  of  theory  construction  and  data 

exploration.  Moreover,  investigators  provide  evidence  for  conceptual  change  resulting 

from  collaborative  interactions.  Apparently  students  critically  appraise  infomiation 

proposed  and  are  persuaded  by  arguments  advanced  by  other  group  members,  resulting  in 

conceptual  change.  Important  observations  reported  but  not  discussed  by  the  authors, 

indicate  students  in  collaboration  tend  to  employ  abductive  reasoning  (generation  of 

tentative  hypotheses  from  best  available  data)  and  inductive  reasoning  (testing  tentative 

hypotheses  with  current  data  and  data  accrued)  rather  than  hypothetico-deductive 

reasoning  (Chamberlin,  1890/1965).  This  matches  with  the  authors'  description  of 

collaborative  process  as  theory  construction,  and  method  of  inquiry  employing  multiple 

working  hypotheses  (Chamberlin)  and  reflective  inquiry  (Dewey,  1896).  For 

Chamberlin, 

.  .  .  under  the  working  hypothesis,  the  facts  are  sought  for  the  purpose  of  ultimate 
induction  and  demonstration,  the  hypothesis  being  but  a  means  for  the  more  ready 
development  of  facts  and  of  their  relationship,  and  the  arrangement  and 
preservation  of  material  for  the  final  induction,  (p.  755) 

This  way  of  knowing  affords  students  the  opportunity  to  consider  equally  all  evidence 

and  to  accept  the  interpretation  most  suggested  by  the  evidence,  or  a  "warranted 

assertion"  pending  better  data  (Dewey).  Implications  for  medical  education  of  this 

method  of  thinking  and  learning  are  important  because  participating  in  such  a  process 
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acknowledges  that  solutions  are  tentative.  This  process  might  facihtate  dialogue  and 
promote  effective  collaboration. 

Clearly,  collaborative  learning  affords  the  potential  to  engage  medical  students  in 
activities  that  are  valuable  in  the  process  of  concept  learning,  and  developing  a 
disposition  toward  intelligent  inquiry.  It  is  not  guaranteed,  however,  that  such  high- 
quality  dialogue  will  transpire  in  a  collaborative  learning  environment.  Medical  student 
collaboration  can  be  characterized  by  competition,  asymmetrical  participation,  and  the 
focus  can  be  on  finishing  a  task  instead  of  understanding  concepts  (Steele,  Medder,  & 
Turner,  2000).  For  example,  Steele  et  al.  observed  that  peer-tutored  groups  "take 
shortcuts  in  the  PBL  process  that  may  undennine  some  of  the  intended  goals  of  PBL"  (p. 
23).  Consequently,  the  learning  process  must  be  verified  to  facilitate  the  valid 
interpretation  of  outcomes.  This  is  discussed  later. 

Tutor  characteristics  and  tutoring  responsibilities.  Ingredients  hypothesized 

as  important  to  fostering  collaborative  learning  during  problem-based  instruction  include 

authenticity  of  the  problem,  the  role  of  the  student,  and  activity  of  the  tutor  (Dolmans, 

Wolfliagen,  Schmidt,  &  van  der  VIeutcn,  1998;  Schmidt  &  Moust,  1995).  For  teachers, 

problem-based  instruction  represents  a  pedagogical  shift  from  a  transmission-oriented 

perspective,  in  which  the  emphasis  is  on  instructors  and  what  they  teach  (instructivism), 

to  an  interaction-oriented  perspective,  in  which  the  emphasis  is  on  students,  and  how  they 

build  knowledge  (constructivism)  (Brown  &  Campione,  1990).  These  concepts  are 

relevant  to  medical  edtication  in  general  and  problem-based  learning  in  particular,  and  are 

expressed  in  the  following  passage  by  Dewey  (1916): 

The  alternative  to  furnishing  ready-made  subject  matter  and  listening  to  the 
accuracy  with  which  it  is  reproduced  is  not  quiescence,  but  participation,  sharing 
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in  an  activity.  Inside  shared  activity  the  teacher  is  the  learner,  and  the  learner  is, 
without  knowing  it,  teacher,  (p.  160) 

Extending  the  Dewey  tradition,  a  tutorial  framework  for  problem-based  teaching 

and  learning  theoretically  renders  each  pupil  both  student  and  teacher  (Federman,  1999). 

This  promotes  group  processing  of  infonnation  rather  than  an  imparting  of  infomiation 

by  a  tutor  (Vemon  &  Blake,  1993).  Such  a  pedagogy  converges  with  Dewey's  (1974) 

belief  that  a  student  cannot  be  taught  what  to  know,  but  can  be  coached. 

He  has  to  see  on  his  own  behalf  and  in  his  own  way  the  relations  between  means 
and  methods  employed  and  results  achieved.  Nobody  else  can  see  for  him,  and  he 
can't  see  just  by  being  'told,'  although  the  right  kind  of  telling  may  guide  his 
seeing  and  thus  help  him  see  what  he  needs  to  see.  (p.  151) 

To  this  end,  quite  possibly  a  PBL  tutor's  greatest  challenge  is  deciding  when  to 
engage  and  when  to  disengage  in  direct  participation  in  PBL  group  activities  (Neville, 
1998).  For  example,  an  important  role  of  the  tutor  is  to  evaluate  the  metacognitive 
competence  of  students  (Norman  &  Schmidt,  1992).  Tutors  face  the  challenge  of 
adapting  their  level  of  engagement  to  their  students'  varied  levels  of  metacognition  in 
ways  that  scaffold  under-developed  self-directed  learning  skills.  Consequently, 
metacognitive  coaching  is  essential  and  metacognitively  challenged  students  must  be 
identified  and  helped  immediately  (Brown,  1994;  Brown  et  al.,  1989),  or  as  mentioned 
earlier,  academic  frustration  and  anxiety  may  manifest. 

As  previously  suggested,  reflective  inquiry  may  be  a  useful  methodology  for  the 
tutor.  Reflective  inquiry  guides  students  away  from  impulsive  thinking  by  encouraging 
dialogue  and  structuring  thinking  about  thinking  (metacognition).  In  this  way,  the  tutor 
encourages  metacognitive  awareness  to  facilitate  self-assessment  to  determine  what  a 
student  or  group  knows  and  does  not  know.  The  basic  challenge  for  the  tutor  is  to 
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develop  in  students  an  interpretive  (interactive)  rather  than  reactive  disposition  (Dewey, 
1933/1998;  Lindblom-Ylanne  &  Lonka,  1999)  that  enables  them  to  perceive  affordances 
(situational  cues)  in  complex,  often  chaotic  and  unpredictable  clinical  problem-spaces  so 
they  can  leant  to  practice  competently  on  the  fringe  of  suiprise  or  in  non-routine 
situations  (Schon,  1987). 

Tutor  activity.  Savery  and  Duffy  (1995)  argue,  "The  most  critical  teaching 
activity  is  in  the  questions  the  tutor  asks  the  learner"  (p.  33).  This  pedagogical  approach 
approximates  the  technique  of  recitation  advocated  by  Dewey  (1933/1998).  Indeed, 
Dewey  indicated  that  the  art  of  conducting  a  recitation  rests  primarily  in  the  art  of 
questioning  learners  to  develop  in  them  independent  disposition  of  inquiry  "in  both  of  its 
directions;  namely  inquiry  in  observation  and  recollection  for  the  subject  inatter  that  is 
pertinent  and  inquiry  through  reasoning  into  the  meaning  of  material  that  is  present"  (p. 
266).  These  questions  aim  to  challenge  the  learner  thinking  and  support  intellectual 
development  via  scaffolding  and  extending  the  zone  of  proximal  development  (ZPD)  as 
described  by  Vygotsky  ( 1 987,  1 926/1 997).  Vygotsky  (1987)  described  ZPD  as  "the 
distance  between  actual  developmental  level  as  determined  by  independent  problem 
solving  and  the  level  of  potential  development  as  determined  through  problem  solving 
under  adult  guidance  or  in  collaboration  with  more  capable  peers"  (p.  86).  Thus,  for 
individuals,  psychological  function  appears  twice,  first  on  a  social  plane  (interpsychical) 
and  then  on  a  psychological  plane  (intrapsychical).  For  Vygotsky  (1926/1997),  an 
individual's  ZPD  represents  boundaries  of  knowing  space,  that  is,  the  gulf  between  actual 
knowledge  or  performance  (e.g.,  independent  problein-solving  competence)  and  potential 
knowledge  or  performance  (e.g.,  guided  problem-solving  competence).  Traditionally 
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these  psychological  functions  (e.g.,  attention,  memory,  and  cognition)  have  been  treated 
as  properties  of  an  individual's  mind.  The  social  dimension  of  psychological  functioning 
proposed  by  Vygotsky  ( 1 978)  helps  explain  how  collaborative  learning  occurs  and 
theoretically  sponsors  efficacy  of  collaborative  learning.  From  this  perspective, 
scaffolding  at  once  strives  to  proximate  ZPD,  while  appropriately  extending  the  upper 
bound  of  ZPD  to  facilitate  the  personal  construction  of  cognitive  representations.  A 
formidable  responsibility  for  tutors  is  to  balance  scaffolding  while  inducing  cognitive 
conflict  to  mediate  knowledge  construction.  Such  a  pedagogical  modus  operandi 
recognizes  that  knowledge  is  perpetually  under  construction  with  titrated  or  "demand 
triggered"  scaffolding  via  a  tutor. 

According  to  Bandura  (1997),  tutors  serve  as  powerful  role  models.  Initially  the 
tutor  is  a  model,  then  a  facilitator.  As  students  develop  expertise,  the  tutor  transitions  to  a 
monitoring  role  and  engages  only  to  insure  quality  of  learning  and  attainment  of  learning 
outcomes  (Barrows,  1996,  2000).  Thus,  an  essential  initial  responsibility  of  the  tutor  is  to 
establish  with  students  a  commitment  to,  and  skills  required  for,  evaluating  their  learning 
and  performance.  Moreover,  a  tutor  models  and  encourages  students  to  evaluate  the 
group's  learning  and  performance.  Eventually,  evaluation  becomes  part  of  what  Dewey 
called  a  complete  act  of  learning  (Dewey,  1933/1998),  as  students  assist  each  other  in 
developing  a  disposition  toward  the  critical  consideration  of  the  quality  of  all  elements  of 
learning  and  its  application. 

Efficacy  of  tutoring.  What  are  the  qualities  of  an  effective  tutor?  Bloom  (1984) 
regarded  individual  tutoring  as  the  gold  standard  of  education  against  which  instructional 
effectiveness  can  be  judged.  Summarizing  data  from  two  of  his  doctoral  students' 
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research.  Bloom  observed  one-to-one  tutoring  produced  an  effect  size  of  2.0. 
Importantly,  Bloom  illustrated  that  most  students  undergoing  one-to-one  tutoring  have 
the  potential  to  achieve  this  effect-size.  Bloom  provides  little  guidance  for  the  process  of 
efficacious  tutoring,  particularly  in  a  manner  employed  for  PBL,  that  is,  group  tutorial. 
In  their  literature  review  of  PBL,  Albancse  and  Mitchell  (1993)  reported  that  students 
preferred  to  be  tutored  by  experts.  In  particular,  expert  tutors  were  observed  to  augment 
students'  ability  to  learn  via  facilitating  identification  of  relevant  learning  issues  and  by 
assisting  in  correcting  gaps  in  knowledge  and  errors  in  processing  infoniiation. 

Dolmans  et  al.  (1998)  revealed  three  elements  of  efficacious  PBL  tutoring 
including  guiding  students  through  the  learning  process,  content  knowledge  input  and 
commitment  to  the  group's  learning.  Schmidt  and  Moust  (1995)  defined  quintessential 
dimensions  of  PBL  tutoring  to  affect  student  learning  as  "'social  congruence"  and 
"cognitive  congruence,"  which  they  posit  are  inextricably  coupled.  Cognitive 
congruence  is  defined  as  a  tutor's  sensitivity  (social  congruence)  to  problem-induced 
difficulties  encountered  by  students.  Wilkerson  (1995)  assessed  ratings  from  first-year 
medical  students  undergoing  PBL  at  Harvard  Medical  School  and  found  four  distinct 
attributes  of  the  highest  rated  tutors.  First,  effective  tutors  did  not  control  the  group  and 
were  able  to  balance  student  directions  with  assistance,  only  intervening  when 
appropriate  to  foster  student  focus  vis-a-vis  problem  context  and  to  facilitate  learning 
(situational  awareness).  Second,  these  tutors  contributed  their  expertise  to  the  group  by 
recommending  resources  and  providing  anecdotal  clinical  experiences.  Third,  these 
tutors  had  the  ability  to  create  a  pleasant  learning  environment  via  appropriately 
responding  to  students.   Fourth,  effective  tutors  stimulated  the  critical  evaluation  of  ideas 
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by  encouraging  students  to  think  beyond  the  obvious  and  to  examine  a  problem  from 
muhiple  perspectives  to  enhance  their  learning. 

Extending  Wilkerson's  (1995)  work,  Dc  Grave  et  al.  (1999)  argue  "tutors  are  not 
tabula  rasa"  (p.  902),  thus  personal  theories  and  perspectives  of  teaching  and  learning 
influence  their  instructional  modus  operandi.  Despite  equivocal  pedagogical  approaches 
to  group  tutoring,  there  appears  to  be  a  relationship  between  a  tutor's  role  and  the  quality 
of  learning  adopted  by  students  (Wilkerson).  Facilitating  through  scaffolding  appears  to 
be  an  essential  role  of  the  tutor  (De  Grave  et  al.),  particularly  because  students  are 
saddled  with  independence  and  ownership  of  their  learning,  which  as  noted  earlier,  may 
be  anxiogenic  and  frustrating  (Albanese  &  Mitchell,  1992;  Barrows,  1986,  1990,  1996, 
2000;  Gijselaers,  1996;  Kaufman  &  Mann,  1998;  Moore  et  al.,  1994;  Norman  &  Schmidt, 
1992;  Vernon  &  Blake,  1992).  Student  competence  in  coping  with  independence  and 
ownership  of  their  learning  may  vary  (De  Grave  et  al.).  Thus,  as  mentioned  previously, 
individualized  goal  directed  scaffolding  might  be  an  important  aspect  of  efficacious 
tutoring  in  PBL  groups. 

De  Grave  and  colleagues  (1999)  wished  to  discover  the  behavior  of  effective  and 
less  effective  tutors.  They  employed  a  Tutor  Intervention  Profile,  a  valid  and  reliable 
instrument  which  quantifies  four-dimensions  of  tutor  behavior.  These  include  stiinulates 
elaboration,  directs  the  learning  process,  stimulates  integration,  and  stimulates  interaction 
and  individual  accountability.  Not  surprisingly,  they  observed  varied  styles  of  tutoring. 
For  instance,  one  pattern  of  tutor  behavior  was  characterized  by  depending  more  on  the 
use  of  expert  knowledge,  compared  to  another  behavior  pattern  that  predominantly  relied 
on  the  ability  to  stimulate  learning  in  a  tutorial  group.  Tutors  who  rated  highest  on  each 
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of  the  four-dimensions  were  perceived  to  be  most  effective  by  students.  Moreover,  tutors 
who  focused  learning  processes  were  perceived  to  be  more  effective  than  those  who 
focused  content.  De  Grave  et  al.  suggest  viewing  four  dimensions  of  effective  tutoring  as 
an  operationahzation  of  scaffolding.  If  scaffolding  is  considered  an  essential  role  of  the 
tutor,  future  research  seems  warranted  to  clarify  the  strengths  and  weakness  of  these 
dimensions  to  facilitate  faculty  development,  reveal  misconceptions  about  teaching  and 
learning,  and  optimize  the  tutorial  process. 

Aligned  with  the  perspectives  of  Vygotsky  (1987,  1926/1997)  and  Bandura  (1986, 
1997),  Lepper,  Drake,  and  O'Donnell-Johnson  (1997)  discovered  effective  tutors  equally 
emphasize  intra-psychical  (cognitive),  inter-psychical  (social),  and  motivational  factors. 
Interaction  of  these  factors  is  illustrated  in  the  observed  behavior  of  effective  tutoring 
(Lepper  et  al.,  p.  122). 

1 .  Expert  tutors  have  content  knowledge  and  topic-specific  pedagogical  knowledge 
to  manage  the  difficulties  learners  encounter,  and  expert  tutors  have  general 
pedagogical  knowledge. 

2.  Expert  tutors  exhibit  high  level  affective  support  and  nurturance  when  interacting 
with  students  (similar  to  the  concept  of  social  congruence  advocated  by  Schmidt 
and  Moust  [1995]  discussed  below). 

3.  Expert  tutors  employ  a  Socratic  style. 

4.  Expert  tutors  are  committed  to  extending  the  intellectual  envelope  of  each  student 
during  each  tutorial. 

5.  Expert  tutors  accomplish  these  activities  via  unimposing  and  indirect  methods 
(e.g.,  via  strategic  use  of  the  Socratic  style). 

6.  Expert  tutors  encourage  reasoned  and  meaningful  thinking  (e.g.,  reflective 
inquiry)  to  foster  self-generated  explanations. 

7.  Expert  tutors  are  exceptionally  committed  to  encouraging  and  motivating 
students. 

How  does  level  of  tutor  expertise  affect  achievement?  Schmidt,  van  der  Arend, 

Moust,  Kokx.  and  Boon  (1993)  studied  1,120  medical  students  in  four  curricular  years, 

1 52  tutors  and  336  tutorials.  Students  in  expert-tutor  groups  spent  more  time  on  self- 
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directed  study  and  exhibited  higher  achievement  scores.  These  findings  were  most 
impressive  for  first-year  students.  A  tutor's  process-facilitation  skills  also  were  related  to 
achievement  scores  and  both  tutor's  knowledge-related  behaviors  and  group-dynamic 
skills  were  correlated,  indicating  that  both  were  essential  elements  of  effective  tutoring. 
In  a  follow-up  investigation,  Schmidt  and  Moust  (1995)  observed  that  subject  matter 
expert  tutors  were  able  to  employ  more  effective  process  facilitative  behaviors  such  as 
asking  stimulating  questions,  offering  counter  examples,  or  seeking  clarification,  and  that 
these  behaviors  were  related  to  achievement,  as  quantified  by  written  test  scores.  They 
concluded  that  to  be  effective,  tutors  must  be  good  facilitators  and  content  experts,  with 
content  expertise  a  pre-condition  of  efficacious  behavior. 

Sobral  (1994)  compared  faculty-tutored  groups  with  peer-tutored  groups  in  a  PBL 
medicine  course  at  the  University  of  Brasilia  and  observed  no  significant  intergroup 
differences  in  either  problem  solving  success  or  student  self-evaluation  of  skills. 
Although,  scores  were  greater  significantly  for  meaningfulness  of  learning  and  usefulness 
of  group  work  in  peer-tutored  groups. 

Dolmans,  Wolfliagen,  and  Schmidt  (1996)  asked  tutors  to  rate  themselves  as 
experts  and  non-experts.  Then  they  examined  data  from  119  tutors  participating  in  135 
tutorial  groups.  Students  were  asked  to  rate  their  prior  knowledge  of  clinical  problems 
and  to  rate  structure  of  information  presented  with  clinical  problems.  Problems  were 
examined  for  magnitude  of  structure.  Student  achievement  was  quantified  using  150 
true-false  questions  at  the  end  of  each  curricular  unit.  Student  achievement  was 
independent  of  tutor  expertise  and  curricular  structure. 
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In  a  similarly  designed  study  using  students  as  non-experts,  Steele  et  al.  (2000) 
compared  achievement  in  student-led  and  faculty-led  problem-based  learning. 
Employing  a  prospective,  Latin-square  crossover  design,  they  investigated  177-second- 
year  medical  students  participating  in  eleven  problems  over  one-year.  For  each  problem, 
half  of  the  students  were  led  by  students  and  the  other  half  by  faculty.  Each  problem 
achievement  was  quantified  as  performance  on  an  examination  incorporating  multiple- 
choice  and  matching  questions  aligned  with  curricular  objectives.  No  differential 
achievement  was  observed  between  students  who  underwent  faculty-led  versus  peer-led 
problem-based  learning.  Data  from  student  evaluations  indicated  they  preferred  peer 
tutoring,  but,  these  same  data  indicated,  and  faculty  observation  confirmed,  instances 
when  peer-tutored  groups  "take  shortcuts  in  the  PBL  process  that  may  undermine  some 
of  the  intended  goals  of  PBL"  (p.  23).  Apparently,  for  students  collaborating  in  short  cuts 
to  truncate  problem-based  learning,  answers  were  more  salient  than  the  process  of 
inquiry.  A  limitation  of  this  study  is  that  all  participating  students  had  a  year  of  faculty- 
tutored  PBL  experience  and  the  cumculum  is  a  hybrid  of  PBL  and  LBL,  thus  students 
may  have  viewed  the  processes  of  PBL  as  less  important  and  ancillary  to  traditional 
didactics  (Steele  et  al.).  This  explanation  is  supported  by  previous  research  conducted  by 
Schmidt  who  revealed  students  entering  medical  school  with  versus  students  without 
tutorial-group  or  self-directed  learning  experience  exhibited  differential  study  strategies. 
This  developmental  nature  of  self-directed  learning  also  is  acknowledged  in  educational 
psychology  literature  (Winne,  1995a,  1995b;  Zimmemian,  1986).  In  addition,  apparently 
the  depth  and  breadth  of  domain  knowledge  affects  capability  and  the  efficacy  of  self- 
directed  learning. 
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Schmidt  ( 1 994)  compared  medical  student  tutors,  non-expert  faculty  tutors,  and 
expert  faculty  tutors  as  a  function  of  medical  student  achievement.  Students  with  low 
prior  knowledge  in  relation  to  unit  content  benefited  from  having  expert  faculty  tutors. 
Tutor  expertise  also  was  important  when  problem-situations  were  ill-structured,  but  was 
independent  of  achievement  when  problems  were  more  structured.  Unexpectedly, 
students  with  low  prior  knowledge  in  an  ill-stmctured  problem  situation  performed 
reasonably  well  having  undergone  peer  tutoring.  Presumably,  peer-tutors  can  effectively 
facilitate  infomiation  translation  to  ill-infomied  students.  This  observation  however, 
requires  additional  research  to  substantiate  and  clarify  the  implications  for  medical 
education.   Particularly  regarding  initial  learning  competence,  the  presence  of  expert 
tutors  diminishes  the  likelihood  of  misconceptions  {Neville,  1998).  However,  Albanese 
and  Mitchell  (1993)  indicate  an  expert  tutor  can  reduce  student  attempts  at  self-directed 
learning  because  experts  tend  to  be  more  directive  during  the  problem-solving  process. 
This  observation  is  substantiated  by  Schmidt  and  Moust  (1995)  who,  using  structural- 
equation  modeling,  found  a  slightly  negative  (-.12)  association  between  tutor  expertise 
and  self-study  time.  This  suggests  an  inverse  relationship  between  a  tutor's  expertise 
contributions  to  group  discussion  and  time  students  spend  in  self-directed  learning.  A 
similar  inverse  effect  has  been  observed  between  tutorial  group  functioning  and  self- 
directed  learning  (Schmidt  &  Gijselaers,  1990).  Taken  together,  these  reports  indicate  an 
inverse  relationship  between  tutorial  group  efficacy  and  required  extra-tutorial  studying. 
Paradoxically,  these  results  support  the  efficacy  of  tutoring  and  small-group  learning,  but 
blunt,  or  obviate  the  aims  of  problem-based  instmction  to  foster  self-directed  learning. 


Undoubtedly,  research  literature  comparing  the  advantages  and  disadvantages 
associated  with  level  of  tutor  expertise  is  contradictory.  EstabUshing  an  appropriate  level 
of  tutor  expertise  requires  fuilher  research;  but,  rather  than  dichotomize  issues,  research 
designed  to  examine  the  differential  efficacy  of  tutor  expertise  vis-a-vis  student 
experience  and  targeted  learning  outcomes  might  be  more  fruitful.  Possibly,  expert  tutors 
and  peer  tutors  provide  uniquely  effective  roles  in  given  contexts.  For  example,  evidence 
presented  earlier  suggests  students  with  low  prior  knowledge  challenged  with  ill- 
structured  problem-settings  may  be  better  served  via  peer  tutoring  (Schmidt,  1994). 
However,  given  the  importance  of  corrective  feedback,  which  requires  some  degree  of 
subject-area  expertise,  serious  questions  remain  regarding  the  use  of  content-experts  as 
tutors. 

Summary.  Situated  learning  theorists  emphasize  the  inherent  benefits  of  working 
together.  Germane  to  medical  education,  collaborative  learning  establishes  conditions 
relevant  to  clinical  practice,  where  students  and  physicians  must  provide  explanations  and 
take  advice.  Theory  and  evidence  suggest  that  infonnation  is  best  retained  for  relevant 
retrieval  in  memory  when  it  is  linked  to  prior  knowledge.  This  process  is  facilitated  by 
cognitive  elaboration  or  reinterpretation  and  re-organization  of  material  (e.g.,  recitation, 
writing  summaries,  and  explaining  synthesis).  Moreover,  peer  discussion  is  one  means  of 
elaboration.  In  peer  discussion,  students  interact  with  material  and  with  each  other, 
generating  more  infomiation  processing  than  when  interacting  with  material  alone.   In 
addition,  tutor  scaffolding  appears  to  influence  significantly  the  quality  of  collaborative 
learning.   Problem-based  learning  environments  immerse  both  students  and  tutors. 
Continual  use  and  refiection  on  the  PBL  processes  broadens  and  deepens  educators" 
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knowledge  and  ability  to  facilitate  teaching.  Optimal  tutoring  may  be  realized  when 
effective  dimensions  of  scaffolding  are  operationalized  (De  Grave  et  al,  1999).  These 
include  stimulating  elaboration,  directing  the  learning  process,  integration  of  knowledge, 
and  stimulating  interaction  and  individual  accoimtability.  An  interesting  and  potentially 
useful  paradox  of  PBL  is  that  students  leam  to  work  collaboratively  (small-group 
tutorials)  while  they  are  developing  as  autonomous  learners  (self-directed  learning),  a 
process  that  incorporates  the  educational  pluralism  advocated  by  Dewey  (1933/1998), 
legitimate  peripheral  participation  (Lave  &  Wenger,  1991),  and  the  social  genesis  of 
individual  understanding  advocated  by  Brown  and  Palinscar  (1989).  hideed,  PBL 
appears  to  be  a  suitable  framework  for  investigating  the  continuum  of  learning  anchored 
with  individual  learning  (endogenous  learning)  and  collaborative  learning  (exogenous 
learning)  (Moshman,  1982). 
Role  of  Self-Directed  Learning 

Medical  educators  are  challenged  to  provide  potential  physicians  with  the  breadth 
and  depth  of  cuiTent  medical  knowledge  and  the  skills  required  to  use  this  knowledge, 
while  simultaneously  aiming  to  produce  physicians  who  can  sustain  and  further  develop 
their  competency  over  a  career  (Mann,  1994).  Compounding  this  challenge  to  medical 
educators  is  the  efficacy  of  current  medical  knowledge  as  characterized  in  the  statement 
by  Dean  Sydney  Burwell  (Harvard  Medical  School)  nearly  a  half-century  ago,  "My 
students  are  dismayed  when  I  say  to  them  half  of  what  you  are  taught  as  medical  students 
will  in  lO-years  have  been  shown  to  be  wrong.  And  the  trouble  is,  none  of  your  teachers 
know  which  half  (cited  in  Pickering,  1956,  p.  1 15).  Consequently,  medical  educators 


aim  to  foster  in  their  students  the  motivation  and  capabihty  to  continue  learning  beyond 
their  formal  training  (Barrows,  2000). 

Developing  self-directed  learning  skills  is  sine  qua  non  to  becoming  an  expert 
learner  in  medical  practice  (Barrows,  1986,  2000;  Schon,  1987).  Indeed,  a  hallmark  of 
intellectual  competence  is  the  ability  to  recognize  personal  knowledge  gaps  and  endeavor 
to  close  them  (Vygotsky,  1987,  1926/1997).  From  a  cognitive  perspective,  self-directed 
learning  is  triggered  when  the  learner  recognizes  that  the  knowledge  demands  of  a 
problem  exceed  personal  knowledge.  The  perceived  knowledge  differential  represents 
potential  stimulation  to  acquire  requisite  knowledge  to  address  problem  solution 
(Zimmennan  &  LeBeau,  2000). 

Self-regulation  in  learning  is  often  related  to  meaningful  learning,  whereas 
external-,  or  other-regulation  is  more  likely  associated  with  knowledge  reproduction  or 
rote  memorization  (Bandura,  1997;  Lindblom-Ylanne  &  Lonka,  1999;  Regan-Smith  et 
al.,  1994;  Winne,  1995a,  1995b).  Conceptually  self-directed  learning  is  very  similar  to 
what  is  called  self-regulated  learning  in  educational  psychology  literature  (Zimmerman  & 
LeBeau,  2000).  Numerous  characterizations  of  self-regulated  learning  permeate  the 
literature  (Bandura;  Winne;  Zimmerman,  1990,  1998).  For  illustration,  social  cognitive 
theory  views  self-regulation  as  incorporating  three  components  including  self- 
observation,  self-judgment,  and  self-reaction  (Bandura).  Bandura  claims  these  processes 
interact  with  one  another  and  influence  a  sense  of  self-efficacy  that  is  required  to  attain 
learning  goals.  Related  activities  are  choosing  or  adopting  learning  goals,  and  self- 
evaluation.  Self-observation  refers  to  a  willingness  to  become  aware  of  one's  behavior 
and  to  cope  with  resulting  emotions.  Self-judgment  calls  for  making  a  choice  between 
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alternative  evaluation  standards.  Self-reactions  can  motivate  students  by  creating  or 
maintaining  positive  feelings  and  expectations  as  well  as  giving  self-rewards. 

Extending  Bandura's  (1997)  position,  Zimmerman  and  Lebeau  (2000) 
synthesized  common  features  among  varied  perspectives  of  self-regulated  learning 
including  student's  selective  use  of  learning  strategies,  responsiveness  to  feedback 
regarding  learning  effectiveness,  and  self-initiated  efforts  to  seeks  out  opportunities  to 
learn.  For  Zimmerman  ( 1 986),  such  learners  are  "metacognitively,  motivationally,  or 
behaviorally  active  promoters  of  their  academic  achievement"  (p.  308).  From  this 
vantage,  self-regulated  learners  employ  three  kinds  of  strategies  to  execute  their  learning 
including  metacognitive  (e.g.,  self-monitoring,  self-evaluating),  motivational  (e.g.,  setting 
goals,  self-reinforcement,  positive  private  talk),  and  behavioral  (e.g.,  scheduling  study 
time,  employing  external  resources). 

Barrows  (1986,  2000)  argues  that  a  potent  stimulus  to  fostering  metacognition  is 
to  have  medical  students  learn  collaboratively  via  problem-oriented  inquiry.  Such  a 
situation  presents  an  opportunity  for  a  learner  to  discover  what  they  do  not  know  and  in 
what  areas  they  must  focus  study.  Theoretically,  this  process  can  foster  a  disposition  to 
identify  learning  issues,  which  in  addition  to  fostering  activation  of  prior  knowledge  and 
elaboration  of  knowledge  (Coles,  1985),  is  thought  to  stimulate  the  development  of  self- 
directed  learning  and  to  promote  lifelong  learning  (Barrows;  Blumberg  et  al.,  1990; 
Walton  &  Mathews,  1 989).  Developing  a  pattern  of  lifelong  learning  is  required  to 
sustain  practice  competence  in  the  continually  changing  field  of  medicine  (Bok,  1984; 
Schon,  1987;  Tosteson,  1994).   Indeed,  because  of  the  medical  information  explosion, 
becoming  an  expert  learner  may  be  the  most  powerful  outcome  of  formal  medical 
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education.  However,  this  outcome  may  not  be  adequately  fostered  in  traditional,  teacher- 
centered  medical  curricula  that  emphasize  rote  memorization,  or  what  is  often  described 
as  "learning  for  school"  rather  than  "learning  for  life". 

Lindblom-Ylanne  and  Lonka  (1999,  2002)  investigated  learning  practices  ("study 
orchestration,"  p.  122)  in  medical  students  enrolled  in  a  traditional  curriculum.  Their 
study  revealed  a  traditional  learning  environment  fostered  misleading  cues  about  what 
and  how  to  study  efficiently  and  forced  many  students  to  study  in  inadequate  ways, 
driving  them  toward  externally  regulated  and  surface  learning.  Study  orchestration  is 
conceptually  defined  as  a  "contextualized  study  approach  adopted  by  individual  students 
or  groups  of  students"  (Meyer,  1991,  p.  297,  cited  in  Lindblom-Ylanne  &  Lonka).  This 
concept  includes  three  dimensions  of  student  learning:  (a)  the  existence  of  qualitative 
individual  differences  in  method  of  engaging  learning  environment  challenges;  (b)  the 
influence  of  context  on  method  of  learning  environment  engagement;  and  (c)  individual 
differences  in  student  perceptions  of  learning  (Meyer).  Thus,  study  orchestration  reflects 
the  interaction  between  the  learner  and  the  learning  environment  and  functionally 
captures  differential  strategies  employed  by  students  as  they  allocate  their  cognitive 
resources  in  a  specific  learning  context.  Operationally,  study  orchestration  may  be 
characterized  as  a  student  exhibiting  either  conceptual  consonance  or  conceptual 
dissonance.  Lindblom-Ylanne  and  Lonka  employed  a  comprehension  monitoring  probe 
(responses  classified  as  knowledge  criteria  or  comprehension  /  application  criteria), 
survey  instrument  (71  items  concerning  learning  approach,  regulation  of  learning,  and 
conception  of  learning)  and  semi-structured  interviews  concentrating  on  learning 
strategies,  study  patterns,  and  perceptions  of  learning  environment.  Stratifying  their 
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results,  investigators  categorized  thirty-five  senior  medical  students  into  four  groups:  (a) 
meaning-oriented  independent  students  (n  =  4;  3  males,  1  female),  (b)  meaning-oriented 
with  novice-like  conception  of  knowledge  (n  =  12;  4  males,  8  females),  (c)  reproduction- 
oriented  and  application-directed  students  (n  =  1 1;  4  males,  7  females),  and  (d) 
reproduction-oriented  and  externally  regulated  students  subdivided  into  two-groups 
including  coherent  (n  =  4  females)  and  dissonant  (n  =  4  females).  Age  (M  =  25.3  years) 
and  gender  distributions  were  statistically  similar  among  groups.  Corresponding  with  the 
institution's  medical  student  population,  69  percent  of  study  participants  were  female. 
Meaning-oriented  independent  students  actively  sought  to  construct  meaning  and 
integrate  infomiation  despite  factually  oriented  assessment  practices.  They  also  exhibited 
metacognitive  awareness  and  conveyed  concise  goals  for  studying.  Similariy,  meaning- 
oriented  students  with  novice-like  conception  of  knowledge  focused  on  understanding 
instead  of  memorization,  and  exhibited  intentional  study  strategies.  Reproductive- 
oriented  and  application-directed  students  emphasized  the  importance  of  acquiring 
knowledge  and  applying  it  later  in  clinical  situations.  These  students,  contrary  to 
meaning-oriented  students,  emphasized  factual  learning  more  than  understanding. 
Reproduction-oriented  and  externally  regulated  students  were  sub-grouped  according  to 
coherence  of  their  study  orchestration,  that  is,  conceptual  consonance  or  dissonance.  In 
the  conceptual  consonance  subgroup,  students  engaged  in  learning  "in  a  'purely' 
superficial  way"  (p.  131)  and  were  rigidly  regulated  by  the  demands  of  a  traditional 
examination  process.  Whereas,  despite  manifesting  some  metacognitive  awareness, 
students  categorized  in  the  dissonance  sub-group  experienced  problems  adapting  to  their 
medical  education  in  general,  and  the  assessment  process  in  particular.  These  students 
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viewed  their  experience  more  as  coping  with  the  learning  environment,  thus  they 
concentrated  on  memorizing  facts,  were  unable  to  integrate  information,  and  had 
problems  in  meaningful  learning.  These  outcomes  clearly  are  undesirable  and  potentially 
hamiful  (Whitehead,  1929).  Importantly,  the  authors  found  meaning-oriented 
independent  students  enjoyed  above-average  study  success,  whereas  reproduction- 
oriented  and  externally  regulated  students  achieved  the  lowest  grades.  The  authors 
concluded  a  traditional  medical  curriculum  creates  an  environment  "in  which  students 
who  manage  not  to  adapt  accordingly  and  who  do  not  become  reproduction-directed  will 
do  well"  (p.  137).  Fortunately,  some  students'  study  practices  are  immune  to  knowledge 
reproduction-orientation  fostered  by  teacher-centered,  assessment  processes  in  a 
traditional  cumculum  (Lindblom-Ylanne  &  Lonka).  Despite  the  relatively  small  sample 
and  idiosyncratic  students  (Finnish  nationals),  these  data  are  comparable  with  other 
reports  (Miller,  1961;  Neame,  1984;  Regan-Smith  et  al.,  1994;  Schmidt,  1983;  Schon, 
1987). 

Taken  together  these  findings  suggest  pedagogical  methods  substantially 
influence  learning  strategies  adopted  by  medical  students.  Evidence  suggests  traditional 
lecture-based  instruction  requires  teachers  to  be  active  and  medical  students  to  be 
passive,  thus  sponsoring  superficial  learning.  Medical  students  are  not  challenged  to 
exercise  critical  thinking,  as  they  are  only  required  to  recall  basic  information.  In 
contrast,  problem-oriented  instruction  requires  students  to  exercise  inquiry  and  personally 
construct  meaningful  knowledge,  and  requires  tutors  to  be  relatively  passive  while 
encouraging  responsibility  for  self-directed  learning. 
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Manifestations  of  self-directed  learning.   What  instructional  conditions 
facilitate  self-directed  learning?  Problem-based  learning  theorists  (Barrows,  1986,  2000; 
Bliiniberg  et  al.,  1990;  Dolmans  &  Schmidt,  1994;  Walton  &  Mathews,  1989)  derive 
insight  for  self-directed  learning  from  adult  learning  theory  (Knowles,  1975,  1984).  For 
Knowles,  self-directedness  depended  on  locus  of  control  for  learning,  that  is,  who  decides 
what  should  be  learned,  who  should  learn  it,  what  methods  and  resources  should  be  used, 
and  how  success  of  effort  should  be  measured.  Therefore,  to  the  extent  the  learner  makes 
those  decisions,  learning  is  considered  self-directed  (Barrows;  Knowles). 

Characterizations  and  operationalization  of  self-directed  learning  are  varied  in 
PBL  literature,  but  in  general,  as  briefed  earlier,  conceptually  they  are  similar  to  what 
cognitive  psychologists  describe  as  self-regulated  learning  (Zimmerman  &  LeBeau, 
2000).  Similar  to  locus  of  control  principle  advanced  by  Knowles  (1975),  Zimmennan 
and  LeBeau  suggest  three  dimensions  or  controlling  influences  of  PBL  on  self-directed 
learning  processes  including  tutor-driven,  group-driven,  and  self-directed.  Self-directed 
learning  processes  are  defined  as  identifying  learning  objectives,  pursuing  learning 
issues,  and  self-evaluating  learning.  Differential  implementations  of  problem-based 
instruction  affect  the  magnitude  of  these  influences  on  self-directed  learning  (Barrows, 
1986,  2000).  That  is,  depending  upon  how  problem-based  instruction  is  implemented, 
defining  what  to  learn  occurs  on  multiple  levels  from  tutor-driven  objectives  to  students' 
collaborafively  or  individually  generated  objectives,  which  affect  varied  degrees  of  self- 
direcfion  (Dolmans  &  Schmidt,  1994;  Zimmennan  &  Lebeau). 

Cognitive  science  research  suggests  that  the  power  of  problem-based  instruction 
to  drive  self-directed  learning  varies  with  the  locus  of  control  (Knowles,  1975)  for 
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generating  learning  issues  spawned  by  the  problem  challenge  and  the  extent  to  which  the 
problem-solver  is  stimulated  to  resolve  identified  knowledge  and  skills  deficiencies  to 
increase  the  likelihood  of  optimal  resolution  of  the  problem  (Barrows,  1986,  2000; 
Dolmans  &  Schmidt,  1994).  When  instruction  is  operationalized  according  to  the 
principles  of  reiterative-PBL,  learning  issues  developed  by  students  are  thought  to  define 
the  entire  content  core  students  need  to  know  (Barrows  &  Tamblyn,  1980). 

Blumbergand  Michael  (1992)  define  learning  issues  as  factual  or  conceptual 
topics  that  tutorial  groups  decide  must  be  better  understood  to  analyze  problem  challenge. 
These  learning  issues  are  assumed  to  guide  individual  study,  and  as  detailed  earlier, 
Dolmans  and  colleagues  (1993)  and  Wendelberger  et  al.  (1996)  both  reported  good 
correspondence  (averages  of  64%  and  90%,  respectively)  between  curricular  objectives 
and  student  generated  learning  issues.  Assuming  the  tutorial  process  is  responsible  for 
structuring  student's  self-directed  learning,  what  elements  of  that  process  are  causal? 

Gijselaers  and  Schmidt  (1990)  hypothesized  ttitorial  group  functioning  would  be 
influenced  principally  by  the  amount  of  prior  knowledge  students  brought  to  the  learning 
situation,  the  quality  of  the  problems  posed,  and  interacfion  between  tutor  and  students 
and  the  learning  situation  in  general.  Further,  functioning  of  the  tutorial  group  would 
influence  individual  student  interest  regarding  problem-focused  topics  and  consequently 
influence  the  amount  of  self-directed  learning  and  eventual  achievement.  Investigators 
designed  a  study  to  test  this  hypothesis.  Variables  of  tutorial  group  functioning  included 
input  (prior  knowledge,  problems,  and  tutor  behavior),  process  (study  time  and  group 
funcfioning),  and  outcome  variables  (achievement  and  interest  in  course  subject  matter). 
Achievement  was  measured  with  an  end  of  course  examination.  Students  were  asked  to 
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rate  problem  quality  using  a  5-point  Likert  scale  for  several  items  (e.g.,  problem  clarity, 
problem  induced  learning  issues)  and  scores  were  averaged  across  items.  The  other 
variables  were  measured  in  a  similar  fashion.  Path  coefficients  for  variables  were 
produced  using  structural  equation  modeling.  (Structural  equation  modeling  is  concerned 
with  testing  theoretical  models  for  the  structure  of  functional  relationships  among 
multivariate  data.)  Investigators  discovered  that  group  functioning  was  moderately 
associated  with  well-constructed  problems  (.43)  and  tutor  functioning  (.31). 
Interestingly,  they  observed  a  negative  association  between  group  functioning  and  self- 
directed  learning  (-.24).  Amount  of  prior  knowledge  (.37)  and  self-directed  learning  (.42) 
were  found  to  influence  achievement.  Amount  of  prior  knowledge  (.20),  group 
functioning  (.57).  and  quality  of  problems  (.16)  influenced  interest  in  subject  matter. 
These  data  indirectly  contmn  the  results  reported  by  Schmidt  and  Moust  (1995)  who 
observed  an  inverse  relationship  between  tutor  expertise  and  self-study  time  (-.12). 

van  den  Hurk,  Dolmans,  Wolfliagen,  and  van  der  Vleuten  (2001)  refined  and 
extended  these  results  using  similar  modeling  techniques  to  investigate  interactions 
between  student-generated  learning  issues,  individual  learning  processes,  individual's 
reporting  in  tutorial  group,  and  achievement.  These  investigators  hypothesized  that  the 
quality  of  learning  issues  influenced  self-directed  study  and  self-directed  study  influenced 
the  pattern  of  an  individual's  participation  in  group  discussion.  First  year  students  (n  = 
195)  were  asked  to  complete  a  questionnaire  reflecting  three  dimensions  of  the  PBL 
tutorial-process  including  quality  of  learning  issues  (1 -item),  self-directed  study  (1 1- 
items),  and  tutorial  group  reporting  (8-items).  Students  were  asked  to  rate  each  item 
using  a  5-point  Likert  scale,  except  quality  of  each  learning  issue  was  rated  using  a  10- 


96 

point  Likerl  scale.  Self-directed  study  items  indicated  the  degree  to  which  students  use 
learning  issues  when  searching  learning  resources  ("learning-issue  driven  approach")  and 
if  students  studied  literature  to  explain  content  to  another  student  by  making  suminaries 
and  notes  ("explanation-oriented  approach").  Tutorial  group  reporting  items  quantified 
the  extent  to  which  novel  infomiation  was  integrated  ("depth  of  discussion")  and  degree 
varied  topics  was  linked  to  problem  ("breadth  of  discussion").  Achievement  was 
measured  using  the  average  score  of  each  group  on  an  end  of  unit  test.  Data  were 
collected  during  weeks  three  (90%  response)  and  four  (88%  responses)  of  a  six-week 
course  involving  a  new  problem  each  week.  Chronbach's  alpha  ranged  from  .64  to  .74 
for  each  factor  except  the  coefficient  for  breadth  of  discussion  was  .53.  The  correlation 
between  the  two  factors  of  reporting  phase  r  =  .66,  p  <  .01 )  was  significant,  indicating 
lack  of  uniqueness.  First  and  second  data  set  values  fit  the  hypothetical  model  with 
varied  success  (p  =  .39  and  p  =  .09,  respectively).  The  model  revealed  the  impact  of 
learning  issues  quality  on  learning  issue  driven  self-study  was  .24.  When  students 
employed  learning  issues  during  individual  study,  they  prepare  in  an  explanation-oriented 
manner  (.51).  An  explanation-oriented  approach  influences  the  depth  of  group  discussion 
(.51),  but  not  the  breadth  of  discussion  (.07).  Higher  achievement  is  fostered  by  the 
depth  of  group  discussion  (.72),  but  not  the  breadth  of  discussion  (-.08). 

Acknowledged  limitations  of  this  study  are  students  rated  quality  of  group 
discussion  and  only  first  year  students  were  modeled.  Notwithstanding  these  limitations, 
these  data  suggest  the  importance  of  learning  issues  for  motivating  self-directed  learning 
and  the  quality  of  subsequent  reporting  to  group.  Moreover,  quality  of  group  discussion 
(depth  and  elaboration)  and  achievement  is  affected  significantly  by  individual 
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preparation  of  group  members,  particularly  when  students  construct  notes  and  summaries 
of  self-directed  learning  ("explanation-oriented  approach").  Individual  reporting  during 
the  tutorial  process  appears  to  benefit  both  individual  and  group  achievement  much  the 
same  as  observed  during  reciprocal  teaching  (Brown  &  Palinscar,  1989)  and  is 
functionally  similar  to  the  dynamics  of  distributed  expertise  operating  within  a 
community  of  learners  (Brown  &  Campione,  1990).  Moreover,  results  of  this 
investigation  support  the  claims  ofboth  cognitive  construction  (Savery  &  Duffy,  1995) 
and  situational  learning  (Brown  et  al.,  1989)  theorists  that  socially  negotiated  knowledge 
benefits  both  the  individual  and  those  collaborating.  Although,  for  knowledge  to  be 
socially  constructed,  students  cannot  just  show  up,  they  must  show  up  prepared  to 
participate  because  "the  essence  of  teamwork  is  pooling  expertise"  (Brown,  1994,  p.  10). 

Clearly,  PEL  affords  students  opportunities  to  experience  knowing  and  learning 
as  socially  mediated  and  constructed  processes.  Moreover,  legitimate  participation  (Lave 
&  Wenger,  1991)  in  a  tutorial-group  process  requires  a  responsibility  beyond  just 
becoming  "expert"  in  a  particular  area.  Students  also  must  contribute  to  (teach)  and  learn 
from  the  distributed  expertise  of  the  group,  because  the  ultimate  assessment  of 
understanding  resides  in  one's  ability  to  transfonn  knowledge  into  teaching  (Shulman, 
1986). 

van  den  Hurk,  Dolmans,  Wolfhagen,  and  van  der  Vleuten  (1999)  also  have 
examined  the  association  of  student-generated  learning  issues,  patterns  of  self-directed 
learning,  and  achievement.  They  designed  a  study  to  discover  if  medical  students  utilized 
learning  issues  as  strict  or  global  guides  for  study,  whether  this  changed  with  year  of 
training,  and  how  generated  learning  issues  related  to  study  time  and  achievement.  A 
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questionnaire  was  developed  incorporating  seven  items  that  measured  to  what  extent 
students  study  strictly  according  to  student-generated  learning  issues  and  six  items  that 
measured  to  what  extent  students  study  beyond  student-generated  learning  issues.  In 
addition,  a  question  regarding  individual  study  time  was  included.  Achievement  was 
measured  using  two  tests,  an  end  of  unit  criterion-referenced  test  and  a  comprehensive 
progress  test.  The  comprehensive  test  sampled  from  total  body  of  medical  knowledge 
and  was  taken  by  all  students  four  times  a  year.  Investigators  assumed  students  cannot 
specifically  prepare  for  progress  tests.  Response  rates  from  medical  students  in  years  1, 
2,  3,  and  4  of  a  six-year  program  were  73%  (n  =  156),  72%  (n  =  149),  72%  (n  =  101),  and 
55%  (n  =  73),  respectively.  The  low  response  rate  for  fourth  year  students  was  attributed 
to  the  fact  that  many  of  them  at  the  time  were  studying  abroad.  However,  sensitivity 
analysis  revealed  the  data  set  for  fourth-year  students  was  similar  to  data  observed  for 
third-year  students.  These  results  indicate  that  first  year  students  tend  to  employ  learning 
issues  as  a  strict  guide  for  self-directed  study;  they  apply  learning  issues  as  a  tool  for 
identifying  specific  information  resources  aligned  with  the  topic  at  hand.  This  strategy 
diminishes  with  academic  development,  as  students  transition  to  employing  learning 
issues  as  a  global  guide  aligned  with  personal  learning  needs  and  interests.  Not 
surprisingly,  students  apparently  become  better  self-directed  learners  in  proportion  to 
their  experience  with  problem-based  instruction.  Importantly,  investigators  observed 
students  who  exercise  global  versus  constrained  approaches  to  self-directed  learning 
spend  more  time  studying  and  realize  better  achievement  on  both  unit  and  comprehensive 
progress  tests.  This  suggests  that  when  students  are  developing  as  self-directed  learners 
in  a  problem-based  instructional  environment,  learning  eventually  is  rewarded  by  success 
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when  undergoing  both  criterion-referenced  and  comprehensive  assessments.  Limitations 
of  this  study  inckide  the  fact  that  the  quahty  and  utihty  of  student-generated  learning 
issues  may  diminish  with  years  of  training,  and  students  who  exceed  learning  issue 
content  are  more  test-driven  rather  than  more  cognizant  of  personal  learning  needs.  This 
latter  limitation  might  be  partially  discounted  because  assessment-driven  learning  can  be 
rewarded  with  criterion-referenced  tests,  but  less  likely  by  a  comprehensive  progress  test. 
Clearly,  progress-assessment  versus  criterion-assessment  strategies  foster  differential 
self-directed  learning  practices.  Progress  assessments  appear  to  establish  an  upper  limit 
of  learning  (e.g.,  zone  of  proximal  development),  thus  challenging  self-directed  learning 
in  a  manner  similar  to  Vygotsky's  (1926/1997)  pedagogical  method. 

As  mentioned  previously,  an  assumption  of  reitorative-PBL  is  that  learning  issues 
generated  by  students  during  problem  discussion  define  the  core  content  students  need  to 
know,  thus  stimulating  and  guiding  self-directed  learning  and  defining  what  constitutes 
tentative  mastery  of  content  (Barrows  &  Tamblyn,  1980).  However,  as  discussed  earlier, 
the  evidence  suggests  that  the  role  of  student-generated  learning  issues  varies  according 
to  the  method  of  implementing  problem-based  instruction  in  general  and  tutor-expertise 
and  group  functioning  in  particular  (Schmidt  &  Moust,  1995). 

Blumberg  et  al.  (1990)  conducted  structured  interviews  of  faculty  in  1987  and 
again  in  1989  at  seven  North  American  medical  schools  to  ascertain  the  institution 
specific  roles  of  student-generated  learning  issues.  The  results  of  these  interviews  were 
framed  as  a  continuum  anchored  by  reiterative  PEL  (University  of  New  Mexico  and 
McMaster  University  where  student-generated  learning  issues  define  all  the  content  to  be 
mastered)  and  faculty-centered  PEL  (Michigan  State  University  and  Mercer  College 
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where  the  role  of  student-generated  learning  issues  is  de-emphasized).  In  the  latter  case, 
tutors  and  students  focus  on  faculty-generated  objectives  and  learning  resources. 
Between  extremes  of  the  continuum,  are  PBL  curricula  that  require  students  to  generate 
learning  issues,  but  also  use  faculty-generated  objectives  (Wake  Forest  University, 
Harvard  University,  Rush  College).  Results  of  this  study  indicate  the  disparate 
approaches  of  PBL  curricula  to  foster  self-directed  learning.  Founders  of  PBL  assumed 
student-generated  learning  issues  were  essential  to  stimulate  and  frame  self-directed 
learning  (Barrows,  1986,  2000).  De-emphasis  of  student-centered  learning  may  obviate 
the  appropriate  development  of  self-directed  learning  skills  and  a  disposition  for  lifelong 
learning  (Blumberg  et  al.). 

This  question  was  subsequently  addressed  by  Blumberg  and  iVIichael  (1992). 
They  surveyed  Rush  IMedical  College  students  in  a  partially  faculty-directed  PBL 
curriculum  (n  =  31,  86%  response)  and  a  LBL  curriculum  (n  =  1 12,  56%  response). 
Although  not  randomized,  students  in  each  curriculum  track  were  obtained  from  the  same 
pool  of  matriculating  students  and  their  educational  goals  were  similar.  They  found  both 
PBL  and  LBL  students  studied  about  25  hours  a  week,  but  study  resources  were 
significantly  different.  Three  resources  employed  most  frequently  by  PBL  students  were 
textbooks,  infomial  discussions,  and  journals,  all  considered  self-directed  learning 
resources.  Lecture-based  students  most  frequently  used  faculty-prepared  course  syllabi 
and  cooperative  lecture  notes,  all  considered  faculty-directed  learning  resources.   Library 
circulation  data  revealed  PBL  students  borrowed  56%  more  books.  This  disparity 
became  more  salient  in  clinical  years,  when  students  who  underwent  PBL  borrowed 
360%  more  books,  despite  both  groups  converging  to  the  same  clerkship  curriculum. 
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Detailed  results  and  analysis  of  this  study  are  presented  in  Chapter  4.  Thus,  despite  some 
de-emphasis  of  student-generated  learning  issues,  PBL  students  still  exhibit  study 
patterns  indicative  of  self-directed  learning;  a  disposition  apparently  sustained  beyond  the 
PBL  curriculum. 

Others  have  conducted  similarly  designed  comparative  studies  and  discovered 
pre-clerkship  PBL  students  at  a  variety  of  medical  schools  exhibit  greater  use  of  library 
resources  than  relatively  matched  cohorts  undergoing  LBL  (Marshall,  Fitzgerald,  Busby, 
&  Heaton,  1993;  McGowan,  1995;  Rankin,  1992). 

Shin,  Haynes,  and  Johnston  (1993)  provide  additional  support  for  sustained  self- 
directed  learning  in  PBL  versus  LBL  students  beyond  formal  curriculum.  They  sought  to 
compare  how  well  graduates  of  a  PBL  versus  a  LBL  curriculum  stay  informed  of  current 
medical  practice.  A  random  sample  of  48  graduates  of  a  PBL  curriculum  (McMaster 
University)  and  48  graduates  of  a  LBL  curriculum  (University  of  Toronto)  stratified  for 
year  of  graduation  (1974  to  1985)  and  gender,  in  family  or  general  practice  in  Ontario, 
agreed  to  be  studied.  A  validated  questionnaire  incorporated  52  multiple-choice 
questions  assessing  management  of  hypertension  was  completed  by  each  participant. 
Results  suggest  graduates  from  McMaster  University,  a  three-year  reiterative-PBL 
curriculum,  are  more  current  in  hypertension  management  knowledge  than  graduates 
from  the  University  of  Toronto,  a  four- year,  LBL  curriculum  (detailed  results  and 
analysis  are  presented  in  Chapter  4). 

Summary.  Problem-based  teaching  and  learning  is  designed  to  encourage 
students  to  develop  self-directed  learning  strategies  by  employing  practice-oriented 
situations  that  require  them  to  confront  their  own  learning  needs  within  the  objectives  of 
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the  faculty.  Operationally,  problem-based  instruction  shifts  the  locus  of  control  from 
instructors  to  students,  employing  methods  linking  learning  process  and  content.  That  is, 
what  is  learned  evolves  from  how  it  is  learned  (Gijselaers,  1996).  This  learning  process 
is  in  stark  contrast  to  traditional,  lecture-based  instruction  where  students  are  taught 
prospectively  determined  facts  and  concepts  independent  of  their  application  to  diverse 
practice  situations.  However,  evidence  suggests  there  is  an  inverse  relationship  between 
self-directed  learning  and  tutor-expertise  and  quality  of  group  functioning. 
Paradoxically,  theory  and  evidence  suggest  immediate  and  accurate  feedback  augments 
initial  learning  and  facilitates  transfer  (Perkins,  1986;  Perkins  &  Salomon,  1989),  but 
fostering  self-directed  learning  is  an  expressed  aim  of  problem-based  instruction.  To 
affect  desired  learning  outcomes  and  reveal  learning  processes,  future  research  is 
warranted  to  examine  the  interaction  between  the  magnitude  of  cognitive  scaffolding  by 
tutor  and  the  locus  of  control  for  learning  issue  generation. 
Role  of  Assessment 

Traditional,  discipline-oriented,  teacher-centered,  or  objectivist  curricula  are 
grounded  in  an  assumed  dualism  between  knower  and  known  (Biggs,  1 996;  Dewey, 
1948,  1933/1998;  Dewey  &  Bentley,  1949;  Schon,  1987).  For  objectivists,  knowledge 
exists  independently  of  the  knower,  and  understanding  is  coming  to  know  what  exists. 
From  this  epistemological  stance,  understanding  is  associated  with  theory  and  application 
is  associated  with  practice  (IMartenson,  1999).  Thus,  teaching  is  the  transmission  of  this 
knowledge,  which  is  decontextualized  by  assessing  it  independent  of  particular  situations. 
Often,  this  pedagogical  approach  primarily  focuses  on  testing  of  basic  knowledge, 
presumably  acquired  via  rote  memorization  of  what  is  presented  in  lecture  or  textbook. 
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In  this  pedagogical  paradigm,  instruction  and  assessment  are  considered  separate 
activities  and  assessment  items  are  in  recognition  and  choice  format,  examining 
decontextuahzed  knowledge  (Biggs;  Perkins,  1986).  Moreover,  in  teacher-centered 
instruction,  assessment  is  designed  to  help  teachers  assess  whether  course  objectives  have 
been  realized,  thus  largely  ignoring  the  needs  of  individual  students.  Consequently, 
standardized  assessments  can  be  employed.  Whereas,  educators  employing  a  social 
constructive  perspective  recognize  "learners  airive  at  meaning  by  actively  selecting,  and 
cumulatively  constructing,  their  own  knowledge  through  both  individual  and  social 
activity"  (Biggs,  p.  348). 

Some  social  constructivists  call  attention  to  the  importance  of  assessmg 
understandmg  and  the  active  use  of  knowledge  (Perkins,  1986);  a  position  advanced  by 
Whitehead  (1929)  who  wrote,  "Education  is  the  acquisition  of  the  art  of  utilization  of 
knowledge"  (p.  4).  If  the  social  constructivist  view  is  accurate  and  learning  has  an 
inherent  constructive  nature,  an  appropriate  pedagogical  stance  requires  a  congruence 
between  curricular  objectives  and  assessments,  otherwise  unintended  outcomes  result 
(Cobb,  2002;  Lindblom-Yliinne  &  Lonka,  1999).  Hence,  efficacious  assessment  must 
transcend  measuring  the  reproduction  of  knowledge  (Biggs,  1996;  Douchy,  Segers,  & 
Sluijsmans,  1999;  Perkins). 

Research  suggests  methods  of  assessment  influence  how  students  study,  despite 
what  teachers  and  syllabi  may  indicate  about  course  objectives.  Thus,  assessment  is 
called  the  "hidden  curriculum"  (Barrows,  1986;  Biggs,  1996;  Dubin  &  Taveggia,  1968; 
Lindblom-Ylanne  &  Lonka,  1999,  2002;  Newble  &  Jaeger,  1983;  ten  Gate,  2001).  This 
perspective  is  succinctly  clarified  by  Norman  ( 1 997)  when  he  wrote  "The  curriculum  tells 
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you  what  the  faculty  are  doing;  the  examination  system  tells  you  what  the  students  are 

doing!"  (p.  265),  and  by  Des  Marchais  and  Vu  (1996)  who  suggested: 

A  student's  learning  evaluation  system  has  so  great  an  effect  on  the  way  students 
dedicate  their  energy  in  learning  medicine  that  any  institution  must  pay  great 
attention  to  developing  an  evaluation  congruent  with  its  curriculum  philosophy, 
its  educational  goals,  and  its  students'  learning  needs,  (p.  282) 

Thus,  what  is  assessed  becomes  valued  by  students  and  guides  their  learning 
strategies.  That  is,  knowledge  is  attached  to  test  cues.  For  illustration,  Lindblom-Ylanne 
and  Lonka  (2002)  studied  perceptions  of  assessment  practices  in  senior  medical  students 
undergoing  a  traditional,  teacher-centered  curriculum  and  noted  that  according  to  the 
students,  the  examinations  too  often  called  for  memorization  instead  of  understanding 
and  applying  knowledge.  Furthemiore,  many  of  the  students  critically  noted  that  the 
examination  practices  guide  students  toward  superficial  learning,  which  is  not  useful  or 
functional  for  their  future  occupation  (p.  136). 

Kelson  (2000)  confirms  and  extends  this  position  by  suggesting  assessment 
shapes  student  perceptions  of  what  it  means  to  know,  what  it  means  to  learn,  and  how 
challenges  are  faced  in  authentic  practice.  Indeed,  when  grades  are  determined  primarily 
by  tests  requiring  recall  of  factual  material,  students  will  focus  much  of  their  cognitive 
energy  in  memorizing  factual  material  (Kelson;  Lindblom-Ylanne  &  Lonka,  2002) 

The  intelligent  use  of  assessments  to  evaluate  retention,  understanding  and  active 
use  of  knowledge  and  skills  would  at  once  diagnose  knowledge  and  skill  gaps  (formative 
role  of  assessment),  while  fostering  understanding  and  what  might  be  called  the 
"generalizability"  of  knowledge  and  problem-solving  expertise  (Douchy  et  al.,  1999; 
Lindblom-Ylanne  &  Lonka,  2002).  Thus,  if  assessment  is  designed  specifically  to  model 
a  student  outcome,  then  cognitive  resources  can  be  expected  to  be  directed  at  that 


105 


outcome  (Bransford  et  al.,  1999;  Lindblom-Ylanne  &  Lonka,  1999;  Thomas,  1997). 
Assessment  then  is  aligned  to  tap  representation  of  content.  This  manner  of  assessment  is 
integrated  with  instruction,  thus  it  becomes  a  tool  for  scaffolding  and  enhancing  learning 
(Shepard,  2000).  Accordingly,  the  locus  of  assessment  is  at  the  center  of  a  cycle  for 
learning  (Shepard).  For  Shepard,  assessment  is  embedded  in  the  process  of  teaching  and 
learning,  that  is  instruction  and  assessment  is  seamless.  In  this  manner,  assessment 
creates  targeted  opportunities  for  teachers  to  gain  insight  about  facilitating  understanding. 
This  pedagogical  approach  is  grounded  in  activity  (Vygotsky,  1987,  1926/1997),  situated 
learning  (Brown  et  al.,  1989;  Greeno,  1998),  and  particularly,  legitimate  peripheral 
participation  (Lave  &  Wenger,  1991).  Moreover,  interactive  assessment  may  facilitate 
the  discovery  of  conditions  of  cognition  (Glasser,  1991 )  and  clarify  the  process  of 
concept  construction  (GaFperin,  1969).  The  implication  for  medical  education  of  an 
interactive  assessment  paradigm  is  that  it  may  reveal  a  critical  threshold  of  biomedical 
science  knowledge  required,  to  provide  sufficient  competence  to  transition  from  content 
novice  to  content  expert,  and  to  orchestrate  efficacious  self-directed  learning  (Winne, 
1995a,  1995b;  Winne  &  Hadwin,  1998).  Furthermore,  if  the  goal  of  medical  education 
research  is  to  measure  the  acquisition  of  instructional  objectives,  further  research  is 
required  to  develop  more  instructionally  sensitive  assessments  (Crocker  &  Algina,  1986) 
to  account  for  what  has  been  learned  in  both  experimental  and  control  medical  instruction 
environments. 

Clearly,  the  view  that  the  assessment  of  medical  student  achievement  is  solely 
somethmg  that  occurs  at  the  end  of  a  process  of  learning  is  no  longer  acceptable  (Douchy 
et  al.,  1999).  Indeed,  traditional  assessment  "presents  a  banier  to  the  implementation  of 
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more  constructivist  approaches  to  instruction"  (Shepard,  2000.  p.  4).  Rather  than 
teaching  to  the  test,  medical  educators  should  teach  with  the  test.  From  this  perspective, 
assessments  can  be  viewed  as  interventional.  This  idea  warrants  investigation,  as 
interventional  assessments  might  prove  efficacious  as  a  learning  tool  while  functionally 
mapping  achievement  as  medical  students  transition  from  concept  novices  to  content 
experts. 

While  not  addressing  assessment  directly,  but  presenting  an  important  philosophy 
for  would  be  assessors,  Dewey  and  Bentley  ( 1 949)  perceived  education  as  the  translation 
of  an  effective  use  of  knowledge  and  achievement  as  effective  use  of  that  knowledge.  In 
a  manner  Dewey  likely  would  endorse,  pioneers  of  PBL  designed  instruction  to  translate 
knowledge  and  skills  in  meaningful  ways  by  immersing  students  in  authentic  contexts 
while  cultivating  problem-solving  skills  and  self-directed  learning  skills;  their  assessment 
process  was  designed  to  assess  the  effective  use  of  this  translated  knowledge,  both 
formatively  and  summatively  (Barrows,  1986,  2000).  Like  other  dimensions  of  PBL, 
assessment  processes  are  at  once  a  philosophy  and  a  method.  Ideally,  problem-based 
instruction  incorporates  assessments  designed  to  reveal  both  how  and  what  learning 
occurred.  In  this  way,  assessment  is  blended  into  the  teaching  and  learning  processes. 
Moreover,  assessment  provides  students  with  opportunities  and  direction  for 
improvement.  From  this  perspective,  assessment  is  inextricably  coupled  with  the 
objectives  of  problem-based  learning  and  employed  to  monitor  student  progress  and 
inform  teaching  practices  (Barrows),  a  functional  relationship  defined  by  Biggs  (1996)  as 
"constructive  alignment."  Therefore,  valid  and  accurate  assessment  of  knowledge 
construction  and  understanding  necessitates  differential  measures  targeting  appropriate 
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knowledge,  skills,  and  attitudes.  For  example,  perfonnance  on  medical  licensing 
examinations  is  necessary,  but  not  sufficient  to  assure  excellence  of  teaching  and  learning 
process  (Blake,  Hosokawa,  &  Riley,  2000). 

Sokas  and  colleagues  (1993)  suggest  PBL  may  have  benefits  that  are  difficult  to 
quantify;  consequently,  some  PBL  programs  have  modified  assessment  towards  a 
reduced  emphasis  on  recall.  Thus,  the  question  remains  an  open  one  as  to  the  continued 
desirability  of  attempts  at  assessment  objectivity.  Moreover,  it  is  important  to  view  PBL 
in  terms  of  its  original  motivations,  as  it  was  not  designed  explicitly  to  satisfy  assessment 
schemes  that  reward  shallow  learning  approaches  (Blake,  Nonnan,  &  Smith,  1995). 

Problem-based  learning  represents  a  paradigm  shift  in  teaching  and  learning  that 
takes  into  account  a  rapidly  changing  knowledge  base  in  medicine,  and  the  requirement 
that  students  become  managers  of  information  and  life-long  learners.  Thus  nontraditional 
types  of  assessment  are  required.  Traditionally  in  medical  education  test  performance 
and  faculty  ratings  have  been  the  dominant  means  of  assessing  students.  These 
traditional  assessment  methods,  however,  are  not  congruent  with  such  curricular  goals  as 
reflective  thinking  and  lifelong  learning.  For  students  to  become  reflective  thinkers  and 
to  develop  self-directed,  life-long  learning  skills,  they  must  be  able  to  appraise  critically 
their  knowledge  strengths  and  weaknesses  versus  their  peers  and  faculty  (Boud,  1992; 
Douchy  et  al.,  1999).  In  such  a  learning  environment,  students  must  be  made  responsible 
for  their  learning.  To  foster  responsibility,  some  educators  recognize  students  must  be 
actively  engaged  in  the  assessinent  process  (Douchy;  Shepard,  2000).  For  example,  in 
small-group  tutorial  settings,  self-assessments  and  peer-assessments  can  be  employed  to 
provide  valuable  insight  regarding  student  performance,  while  engaging  students  in 
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desired  outcomes  of  PBL  (e.g.,  self-assessment  and  critical  appraisal)  (Das,  Mpofu, 
Dunn,  &  Lanphear,  1998;  Savery  &  Duffy,  1995).  Self-assessment  refers  to  learners 
making  judgments  about  their  learning  (Boud,  1995),  an  activity  which  is  typically 
employed  for  formative  assessment  to  facilitate  reflection  on  one's  learning  processes 
and  results  (Schon,  1987).  Self-assessments  have  been  observed  to  correlate  well  with 
tutor-assessments  of  medical  students  undergoing  small-grotip  PBL  (Das  et  al.). 

Another  assessment  innovation,  "triple-jump,"  was  developed  al  McMaster 
University.  Triple-jump  is  a  three-stage  written  or  oral  challenge  that  assesses  a  student's 
ability  to  analyze  and  resolve  a  clinical  problem  or  case  using  self-directed  learning  skills 
(Painvin,  Neufeld,  Norman,  Walker,  &  Whelan,  1979;  Smith,  1993).  In  contrast  to 
traditional  assessments,  triple-jump  is  designed  to  assess  learning  processes.  Triple-jump 
assessment  appears  as  a  means  to  evaluate  individuals  as  they  simulate  what  they  do  in  a 
PBL  tutorial  group.  According  to  Smith  (p.  210),  students  are  assessed  for  their 
performance  at  each  of  the  following  stages: 

1 .  The  student  is  presented  with  a  new  scenario  or  case  that  requires  him/her  to 
identify  the  main  problems  or  issues  in  his/her  own  words.  They  also  identify 
early  hypotheses  or  key  questions  at  this  stage  and  their  initial  approach  to  the 
task. 

2.  Students  are  then  expected  to  find  suitable  data  from  a  variety  of  resources,  to 
check  their  reliability  and  validity,  and  to  interpret  the  data  in  relation  to  their 
hypotheses  or  key  questions. 

3.  Students  then  apply  their  findings  to  resolve  or  inanage  the  problem,  or  to  address 
their  key  questions  and  then  evaluate  how  they  approached  the  task,  and  what 
they  would  do  differently  next  time. 

From  this  assessment  perspective,  a  medical  student's  performance  of 

understanding  would  include  metacognitive  understanding,  reflective  inquiry  (problein- 

solvmg  ability),  and  evaluation  of  decisions  relative  to  the  science  that  informs  medicine 

(the  integration  of  theory  and  practice).  These  are  the  skills  which  can  improve  decision 
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making  and  generate  new  approaches  to  practice.  This  view  echoes  Schon  (1987)  who 
envisioned  cognitive  monitoring  and  decision  malcing  as  mechanisms  linking  knowledge 
and  action,  or  what  might  be  described  as,  clinical  praxis  (Bawden,  1997)  or  pragmatic 
constructivism  (Cobb,  2002). 

Performance  of  understanding  warrants  authentic  assessment.  Assessment  framed 
within  a  clinical  context  encourages  students  to  appreciate  inter-relationships  among 
disciplines  and  topics  (Schon,  1987).  Theory  suggests  this  should  improve  long-term 
memory  and  retrieval  skills.  Moreover,  authentic  assessment  challenges  a  medical 
student  to  perfonn  in  a  manner  similar  to  clinical  practice.  Such  integrated  and  authentic 
assessment  requires  students  to  employ  specific  content  in  a  manner  that  demonstrates 
learning  at  an  extended  abstract  level,  that  is,  they  must  perfonn  their  understanding 
(Dewey,  1933/1998).  Employing  the  triple-jump  framework  appears  a  useful  means  to 
discover  whether  higher  order  objectives  of  PBL  are  being  achieved.  Moreover,  an 
important  impHcation  of  the  triple-jump  framework  for  PBL  is  an  emphasis  on  self- 
reflection  and  self-assessment,  each  important  ingredients  of  efficacious  self-directed 
learning.  The  strengths  and  weaknesses  of  triple-jump  assessment  process  to  problem- 
based  teaching  and  learning  warrant  empirical  investigation. 

Summary.  Students  often  respond  to  "hidden  curriculum"  and  rather  than 
develop  problem-solving  strategies  and  skills,  they  tend  to  focus  their  energy  on  the 
demands  of  assessment.  That  is,  traditional  teaching  and  learning  cultivates  in  students  a 
modus  operandi  designed  to  achieve  "point  in  time  competence,"  i.e.,  a  mastery  learning 
model.  Although,  point  in  time  competence  is  necessary,  it  is  not  sufficient  for  sustained 
practical  competence.  Evidence  suggests  assessment  is  central  to  the  effectiveness  of 
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medical  education.  Research  outcomes  indicate  methods  of  assessment  not  only 
influence  student  conceptions  of  learning,  but  also  student  approaches  to  learning. 
Medical  students  modify  their  approach  to  learning  predicated  on  their  perception  of  how 
information  presented  will  be  assessed.  When  assessment  is  congruent  with  curricular 
objectives,  assessment  appropriately  communicates  what  should  be  learned.  A  mismatch 
between  assessment  and  curricular  objectives  may  direct  student  preparation  and  learning 
inappropriately.  For  example,  assessments  grounded  in  material  presented  lecture-by- 
lecture  fail  to  assess  the  integration  of  information  across  the  content  of  course. 
Assessment  items  framed  within  a  clinical  context  encourage  students  to  appreciate  inter- 
relationships among  disciplines  and  topics.  Moreover,  theory  suggests  this  should 
improve  long-term  memory  and  retrieval  skills  (Brown  et  al.,  1989;  Savery  &  Duffy, 
1995;Schon,  1987). 

I  mplications  Leading  to  the  Present  Study 
Problem-based  instruction,  a  pedagogical  echo  from  Classical  Greece, 
operationalizes  contemporary  principles  of  learning  and  Dewey's  conception  of  reflective 
inquiry.  However,  despite  predictions  of  unique  superiority,  the  relative  efficacy  of 
problem-based  versus  traditional  lecture-based  instructional  approaches  in  medical 
students  and  physicians  is  contradictory  (Albanese  &  Mitchell,  1993;  Berkson,  1993; 
Colliver,  1999;  Vernon  &  Blake,  1993).  Consequently,  some  medical  educators  have 
recommended  caution  when  considering  conversion  to  problem  based  learning  (Albanese 
&  Mitchell;  Berkson;  Colliver;  D'Eon,  1999;  Morgan,  1995).  Since  many  medical 
schools  are  currently  contemplating  significant  cumcular  revision,  the  efficacy  of 
problem-based  instruction  is  crucial  (D'Eon;  Harden  et  al,  2000;  Kaufman,  2000; 
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Prideaux,  2000;  Rothman,  2000).  Disadvantages  reported  include  inferior  biomedical 
science  knowledge  exhibited  by  PBL  versus  LBL  students  as  evidenced  by  the  resuUs  of 
licensing  examinations.  In  addition,  some  evidence  suggests  that  compared  to  their 
traditional  counterparts,  PBL  students  exhibit  greater  academic  stress  and  perceive  that 
they  acquire  less  content  knowledge.  Cited  advantages  include  better  self-directed 
learning  skills  and  more  favorable  attitudes  toward  learning,  although  these  observations 
are  not  unifonn  across  studies.  Therefore,  it  is  logical  to  conduct  the  present  study  using 
meta-analytic  methods  to  examine  the  relative  effect  of  problem-based  versus  lecture- 
based  instruction  on  biomedical  science  achievement,  clinical  science  achievement, 
problem-solvmg  skills,  self-directed  learning  and  a  positive  attitude  toward  learning. 


CHAPTER  3 
METHODOLOGY 

The  purpose  of  this  study  was  to  investigate  whether  problem-based  versus 
lecture-based  teaching  and  learning  affects  superior  cognitive  and  non-cognitive 
educational  outcomes  in  medical  students  and  physicians.  This  chapter  describes  the 
meta-analytic  methods  used  to  compare  the  relative  efficacy  of  problem-based  versus 
lecture-based  teaching  and  learning  for  affecting  biomedical  science  achievement, 
clinical  science  achievement,  problem-solving  achievement,  self-directed  learning  skills, 
and  positive  attitudes  toward  learning.  Initially  testable  hypotheses  for  questions  posed 
by  this  research  are  listed.  The  balance  of  the  chapter  describes  methods  employed  to 
sample  research  studies,  code  dependent  and  independent  variables,  compute  effect  size 
estimates,  analyze  data,  and  examine  threats  to  (or  "questions  of)  reliability  and  validity. 
Rationale  for  a  Meta-Analysis 

Accumulation  is  the  hallmark  of  science  (Kuhn,  1996),  and  without  synthesis,  the 
application  of  research  is  limited  (Dewey  1974;  Dubin  &  Taveggia,  1968;  Lipsey  & 
Wilson,  1993).  For  Dewey,  the  "science  of  education  depends  on  systematic 
accumulation"  (p.  181).  As  the  body  of  published  research  in  any  domain  expands,  there 
becomes  a  need  for  synthetic  analysis  of  work  completed.  Secondary  analysis  of 
cumulative  primary  research  was  probably  introduced  by  Pearson  (1904)  when  he 
averaged  estimates  from  five  independent  samples  of  the  association  between  typhoid 
fever  vaccination  and  mortality  in  soldiers  of  the  British  Army.   Dubin  and  Taveggia 
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(1968)  may  have  been  the  first  to  conduct  and  publish  a  secondary  analysis  of  educational 
research.  Glass  (1976)  is  credited  with  coining  the  term  "meta-analysis."  Compared  to 
independent  research,  meta-analysis  increases  power  and  precision,  provides  an  overall 
estimate  and  range  of  effect,  and  identifies  variability  among  study  results,  which  may 
facilitate  the  discover  of  important  effect  moderators  (Cooper,  1998).  Meta-analysis  is  a 
systematic  method  incorporating  statistical  analyses  for  extracting,  comparing,  and 
combining  results  from  independent  studies  to  quantify  outcomes  that  are  generalizable 
(Cooper).  In  particular,  the  results  of  this  study  will  be  examined  to  discover  common 
conclusion  patterns  in  the  aggregated  data  sets  of  independent  studies  comparing  the 
efficacy  of  problem-based  versus  lecture-based  instruction  from  varied  perspectives 
employing  varied  methodologies.  In  this  mamier,  meta-analytic  methods  mirror 
qualitative  methods  of  triangulation,  which  is,  converging  on  a  "true"  value  by  multiple 
methods  (Campbell  &  Fisk,  1959). 

Meta-analytic  modeling  also  can  detect  and  promote  examination  and  explanation 
of  variations  between  studies.  For  example,  meta-analysis  can  facilitate  the  identification 
of  associations  between  methodological  or  substantive  characteristics  and  outcomes, 
which  is  not  possible  in  single  invesfigations.  Theoretically  this  method  improves 
statistical  power  and  enables  the  expression  of  the  degree  to  which  results  reflect  varied 
measures  of  the  same  construct  (Cooper,  1998).  Meta-analysis  also  can  identify  gaps  in 
knowledge  as  revealed  in  published  literature,  thus  providing  guidance  for  future 
research. 
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Hypotheses 

The  following  hypotheses  guided  this  research: 

Hypothesis  1 :  Aggregated  effect  size  estimates  for  biomedical  science  achievement  in 
students  and  physicians  who  undergo  medical  education  with  problem- 
based  versus  lecture-based  teaching  and  learning  will  be  significantly 
positive  because  the  lower  limit  of  a  95%  confidence  interval  will  exceed 
zero. 

Hypothesis  2:  Variance  of  aggregated  effect  size  estimates  for  biomedical  science 

achievement  will  exceed  random  error  as  evidenced  by  a  Q-statistic  with  a 
E  <  .05  warranting  subgroup  analyses  to  test  for  potential  moderation  by  a 
priori  defined  outcome  measures,  methodological  features,  and 
substantive  features  of  problem-based  teaching  and  learning. 

Hypothesis  3:  Aggregated  effect  size  estimates  for  clinical  science  achievement  in 

students  and  physicians  who  undergo  medical  education  with  problem- 
based  versus  lecture-based  teaching  and  learning  will  be  significantly 
positive  because  the  lower  limit  of  a  95%  confidence  interval  will  exceed 
zero. 

Hypothesis  4:  Variance  of  aggregated  effect  size  estimates  for  clinical  science 

achievement  will  exceed  random  error  as  evidenced  by  a  Q-statistic  with  a 
E  <  .05  warranting  subgroup  analyses  to  test  for  potential  moderation  by  a 
priori  defined  outcome  measures,  methodological  features,  and 
substantive  features  of  problem-based  teaching  and  learning. 

Hypothesis  5:  Aggregated  effect  size  estimates  for  problem-solving  achievement  in 
students  and  physicians  who  undergo  medical  education  with  problem- 
based  versus  lecture-based  teaching  and  learning  will  be  significantly 
positive  because  the  lower  limit  of  a  95%  confidence  interval  will  exceed 
zero. 

Hypothesis  6:  Variance  of  aggregated  effect  size  estimates  for  problem-solving 

achievement  will  exceed  random  error  as  evidenced  by  a  Q-statistic  with  a 
E  <  .05  warranting  subgroup  analyses  to  test  for  potential  moderation  by  a 
priori  defined  outcome  measures,  methodological  features,  and 
substantive  features  of  problem-based  teaching  and  learning. 

Hypothesis  7:  Aggregated  effect  size  estimates  for  self-directed  learning  in  students  and 
physicians  who  undergo  medical  education  with  problem-based  versus 
lecture-based  teaching  and  learning  will  be  significantly  positive  because 
the  lower  limit  of  95%  confidence  interval  will  exceed  zero. 
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Hypothesis  8:  Variance  of  aggregated  effect  size  estimates  for  self-directed  learning  will 
exceed  random  eiror  as  evidenced  by  a  Q-statistic  with  a  g  <  .05 
warranting  subgroup  analyses  to  test  for  potential  moderation  by  a  priori 
defined  outcome  measures,  methodological  features,  and  substantive 
features  of  problem-based  teaching  and  learning. 

Hypothesis  9:  Aggregated  effect  size  estimates  for  positive  attitudes  toward  learning  in 
students  and  physicians  who  undergo  medical  education  with  problem- 
based  versus  lecture-based  teaching  and  learning  will  be  significantly 
positive  because  the  lower  limit  of  a  95%  confidence  interval  will  exceed 
zero. 

Hypothesis  10:  Variance  of  aggregated  effect  size  estimates  for  positive  attitudes  toward 
learning  will  exceed  random  error  evidenced  by  a  Q-statistic  with  a  g  <  .05 
warranting  subgroup  analyses  to  test  for  potential  moderation  by  a  priori 
defined  outcome  measures,  methodological  features,  and  substantive 
features  of  problem-based  teaching  and  learning. 

Candidate  Literature  Criteria 

For  the  period  January  1 966  through  January  2002,  published  and  unpublished 
primary  research  comparing  problem-based  learning  (PBL)  and  lecture-based  learning 
(LBL)  was  identified  by  searching  electronic  databases  including  Academic  Search 
Premier,  Applied  Science  &  Technology  Abstracts,  Academic  Index,  Contents  First, 
Current  Contents,  Educational  Abstracts,  Education  Resources  Infomiation  Center,  First 
Search  Plus,  General  Science  Abstracts,  MedLine,  PsycLit,  Psych  Info,  Reader's  Guide 
Abstracts,  Sociological  Abstracts,  Wilson  Select  Plus;  published  literature  reviews, 
bibliographies  of  primary  research,  and  manual  inspection  of  journals  likely  to  publish 
such  research  including  Academic  Medicine.  Advances  in  Health  Sciences  Education- 
Journal  of  Medical  Education.  Medical  Education.  Medical  Teacher,  and  Teaching  and 
Learning  in  Medicine.  Reports  were  selected  if  they  compared  PBL  and  LBL  for  medical 
students  or  physicians  and  provided  sufficient  quantitative  information  to  compute  effect 
size  estimates.  Although  some  studies  did  not  contain  sufficient  data  to  compute  effect 
size  esfimates,  authors  provided  narrative  regarding  direction  of  instructional  effect,  for 
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example  "a  positive  mean  difference,"  "a  negative  mean  difference,"  or  "no  relationship 
in  either  direction."  Sometimes  these  directional  effects  were  described  as  "significant" 
or  "not  significant,"  but  without  specifying  level  of  significance  or  sample  size.  Some 
meta-analysts  employ  the  sign  test  ("vote-counting"  method)  to  examine  such  data 
(Lipsey  &  Wilson,  2001 ).  However,  since  vote  counting  method  offers  no  way'of 
determining  power,  all  studies  are  usually  counted  equally.  Consequently,  vote  counting 
is  biased  toward  "no  effect,"  when  in  fact  it  is  not  known  if  there  was  no  effect,  or  merely 
no  statistically-discernible  effect  given  sample  size  and  context  (Lipsey  &  Wilson). 
Therefore,  vote-counting  was  not  employed  for  this  research. 

IVIethodological  Design 

Meta-analysis  is  a  method  of  statistically  aggregating  results  of  independent 
investigations  examining  similar  outcomes  (Glass,  1976).  An  assumption  of  meta- 
analytic  methodologies  is  eligible  research  studies  are  viewed  as  a  population  to  be 
systematically  sampled  (Cohen,  1988;  Flciss,  1993).  Although  methodological  adequacy 
varies  between  investigations  in  aggregated  sets,  standards  for  design  quality  are  not 
unifonn  and  no  evidence  exists  to  support  a  study  of  quality  and  effect  size  correlation 
(Glass,  1977;  Lipsey  &  Wilson,  2001;  Wortman,  1994).  Glass  (1977)  argues  "there  are 
no  perfect  studies,  but  it  is  reasonable  to  believe  that  together  imperfect  studies  can 
converge  on  a  legitimate  conclusion"  (p.  356).  Campbell  (1986)  describes  this 
methodological  perspective  as  "proximal  similarity"  (p.  68). 

As  suggested  previously,  this  approach  also  is  similar  to  the  principle  of 
triangulation  advanced  by  Campbell  and  Fisk  (1959).  Triangulation  draws  inferences 
from  multiple  methods  about  processes  operationalzing  a  construct.  From  this 


117 

perspective,  considering  all  available  primary  research  comparing  problem-based  and 
lecture-based  learning  independent  of  methodological  quality  is  important  because  meta- 
analysis can  reveal  whether  methodological  or  substantive  variations  in  primary  research 
studies  are  associated  with  variation  in  effect  sizes  (Glass,  1977).  In  addition,  this 
method  examines  the  extent  to  which  such  information  is  reported  adequately  in  medical 
education  research,  and  provides  an  overview  of  methodologies  from  which  to  define 
characteristics  desirable  in  future  research  (Lipsey,  1 994). 
Independent  Variables 

The  primary  independent  variable  for  this  study  is  instructional  technique. 
Included  investigations  quantified  the  differential  effects  PBL  versus  LBL  with  regard  to 
biomedical  science  achievement,  clinical  science  achievement,  problem-solving 
achievement,  self-directed  learning  skills,  and/or  attitude  toward  learning  in  medical 
students  or  physicians.  Informed  by  educational  philosophy,  principles  of  infonnation- 
processing,  social  constructivism  and  situated  learning  as  detailed  in  Chapter  2, 
secondary  independent  variables  potentially  responsible  for  variation  in  effect  size 
estimates  were  coded  to  facilitate  analyses  of  outcome  features  (outcome  source,  outcome 
type);  methodological  features  (sampling  technique,  baseline  equivalence,  control  group, 
longitudinal  design,  publication  source,  publication  year);  and  substantive  features  (scope 
of  instruction,  institutions'  PBL  experience,  students'  PBL  experience,  PBL  taxonomy, 
problem  structure,  learning  issues  responsibility,  tutor  expertise,  and  self-directed 
learning  responsibility).   Such  explicit  a  priori  coding  directs  an  appropriate  examination 
of  potential  moderators  of  effect  size  estimates  while  reducing  the  temptation  to  data 
dredge  ("fishing")  and  avoids  producing  spuriously  significant  results  by  capitalizing  on 
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chance  (Cohen,  1988;  Lipsey,  1994;  Stock,  1994).  High  inference  coding  of  independent 
variables  is  employed  when  study  features  are  not  included  in  a  report,  but  included  in 
other  temporally  comparable  reports  by  the  same  investigator,  or  other  investigators  from 
the  same  institution  (Cooper,  1998).  Coded  variables  were  entered  into  an  Excel  2002 
(Version  10.2,  Microsoft  Corporation,  Redmond,  WA)  database. 
Dependent  Variable 

The  dependent  variable  is  size  of  effect  when  PBL  and  LBL  were  compared  with 
regard  to  biomedical  science  achievement,  clinical  science  achievement,  problem-solving 
achievement,  self-directed  learning,  and/or  attitude  toward  learning.  Throughout  this 
dissertation  the  symbol  ES  will  refer  to  standardized  effect  size  (Lipsey  &  Wilson,  2001 ). 
Metric  for  Estimating  Effect  Size 

Meta-analysis  requires  quantifying  the  results  of  each  study  in  the  form  of  a 
measure  of  magnitude  of  effect  in  that  study,  or  "effect  size"  expressed  using  a  common 
metric.  Effect  size  values  above  zero  indicate  experimental  intervention  (PBL)  had  a 
positive  effect  on  outcomes;  values  below  zero  indicate  a  negative  effect,  or  relative  merit 
of  LBL.  The  method  for  computing  effect  size  was  guided  by  the  data  reported.  Raw 
data  were  transfomied  into  effect  sizes  with  MetaWin  (Version  2.02,  Sinaurer  Associates, 
Sunderland,  MD)  using  the  computations  described  below.  When  numerical  data 
required  for  effect  size  calculations  only  were  presented  graphically,  the  image  was 
scanned.  The  image  then  was  imported  into  Sigma  Scan  (Version  2.0,  SPSS,  Chicago, 
IL)  to  provide  a  Cartesian  coordinate  model  for  a  precise  estimate  of  data  points  (e.g.,  M 
±  SD).  The  basic  metric  used  to  estimate  effect  size  was  the  mean  of  the  PBL  group 
minus  the  mean  of  the  LBL  -  group  divided  by  a  pooled  standard  deviation  (Cohen  1988). 
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A  pooled  standard  deviation  was  employed  to  compute  effect  size  because  typically  an 
assumption  of  homogeneity  of  variance  in  the  population  is  reasonable,  thus,  a  pooled 
standard  deviation  is  more  stable  and  provides  a  better  estimate  of  population  variance 
than  a  control  group  standard  variation  (Cohen,  1988;  Rosenthal,  1994).  Moreover, 
effect-sizes  estimated  using  data  other  than  group  mean  and  standard  deviation  are  more 
appropriately  comparable  to  those  computed  einploying  a  pooled  standard  deviation 
(Cohen;  Rosenthal).  For  example,  achievement  gain  frequently  is  quantified  as  a 
proportion  of  correct  responses.  Effect  size  for  differences  between  proportions  exhibited 
by  two  groups  was  estimated  by  treating  each  proportion  as  the  mean  distribution  of  O's 
and  I's  (McNemar,  1962).  This  estimate  is  robust  for  even  zero  proportions,  unless  more 
than  70%  of  the  sample  has  a  proportion  of  zero  (Rosenthal).  Estimates  of  effect  size 
from  data  in  such  fomis  as  t  tests  (equal  or  unequal  sample  sizes),  F  tests  (equal  or 
unequal  sample  sizes),  p  levels,  z  scores,  and  r  values  were  computed  with  conversion 
formulae  described  by  Glass,  McGaw,  &  Smith  (1981)  and  Rosenthal. 

Effect  size  estimates  for  each  main  effect  were  screened  using  Statistica  (Version 
6.0,  StatSoft,  Tulsa,  OK.)  notched  boxplot  method  (Hedges  &  OIkin,  1985;  Hedges, 
1994).  Outliers  were  defined  as  effect  size  estimates  distributed  more  than  1.5  box 
lengths  from  the  upper  or  lower  edge  of  the  box  given  a  box  length  equal  to  interquartile 
range.  Outliers  were  examined  for  coding  errors.  After  ruling  out  coding  error,  study 
features  of  outliers  were  compared  with  those  of  included  studies  to  examine  common 
and  idiosyncratic  patterns  and  then  critically  analyzed  via  narrative  in  Chapter  4.  A 
sensitivity  analysis  (Lipsey  &  Wilson,  2001 )  was  performed  to  examine  the  quantitative 
affect  of  excluding  outliers  from  the  mean  effect  size  estimate  for  involved  outcomes. 


120 

For  Lipsey  and  Wilson,  if  sensitivity  analysis  does  not  materially  change  results, 
confidence  in  results  is  strengthened.  If  results  are  altered  in  a  way  that  inight  lead  to 
different  conclusions,  this  indicates  the  need  for  caution  in  interpreting  results  and 
drawing  conclusions. 
Computation  of  Mean  Effect  Size 

Assumptions  underlying  meta-analysis  include  set  of  studies  undergoing  synthesis 
represent  the  population  of  possible  studies  (Cohen,  1988;  Fleiss,  1993)  and  effect  size 
estimates  in  different  studies  are  statistically  independent  (Cohen;  Rosenthal,  1994). 
However,  a  problem  emerging  from  aggregating  effect  sizes  is  deciding  what  constitutes 
an  independent  estimate  of  effect  size  when  a  single  investigation  reports  multiple 
outcomes  (Rosenthal).  When  a  single  investigation  included  multiple  effect  sizes  (i.e., 
multiple-end  points)  for  the  same  outcome,  redundant  effect  sizes  were  coded 
individually.  Aggregating  nonindependent  data  can  exacerbate  Type  1  error  rates  (Cohen; 
Rosenthal),  whereas  discarding  data  would  have  the  opposite  effect,  increasing  the 
likelihood  of  committing  a  Type  II  error.  Consequently,  multiple  effect  size  estiinates  for 
the  same  outcome  in  a  single  study  were  averaged  to  compute  effect  size  estimates 
(Lipsey  &  Wilson,  2001)  and  are  reported  separately  for  each  outcome  category- 
biomedical  science  achievement,  clinical  science  achievement,  problem-solving 
achievement,  self-directed  learning,  and  attitude  toward  learning.  Thus,  a  study  will 
contribute  only  one-effect  size  proportionate  to  sample  size  for  an  outcome  category. 
Different  studies  estimate  the  effect  size  with  various  precision.  That  is,  effect  sizes  often 
are  derived  from  varied  sample  sizes.  Thus,  they  exhibit  different  sampling  errors.  A 
samphng  error  is  defined  as  a  disparity  between  a  sample  characteristic  and  a  population 
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characteristic  (Crocker  &  Algina,  1986).  Sampling  errors  exhibit  predictable 
characteristics  including:  the  mean  of  sampling  errors  is  zero;  sampling  errors  are 
nonnally  distributed  around  an  observed  mean;  the  larger  the  standard  deviation  of  a 
population,  the  larger  the  distribution  will  be  in  the  sample  mean  due  to  a  greater  amount 
of  sampling  error;  and  the  smaller  the  sample,  the  larger  the  sampling  error  will  be 
(Cooper,  1998;  Hunter  &  Schmidt,  1994).  To  control  for  the  difference  between  large 
and  small  sample  sizes  when  aggregated,  a  weighted  mean  of  effect  sizes  in  each  data  set 
was  calculated  to  give  more  weight  to  more  precise  estimates.  From  a  statistical 
standpoint,  this  technique  affords  more  weight  to  studies  with  larger  samples  (Glass, 
1979;  Glass  et  al.,  1981 ;  Hedges,  1994).  Therefore,  effect  size  estimates  were  weighted 
as  the  inverse  of  an  associated  variance  using  fonnulae  described  by  Hedges  and  Olkin 
(1985).  Uncertainty  in  effect  size  value  is  quantified  with  a  95  percent  confidence 
interval  (Cooper).  The  confidence  interval  includes  the  range  of  values  with  95% 
probability  of  including  population  effect  size.  If  the  confidence  interval  does  not  contain 
zero,  the  cumulative  efTect  size  value  is  significant  at  the  .05  level. 
Data  Analyses 

An  important  characteristic  of  meta-analytic  methods  is  the  variance  analysis  of 
effect  size  estimates  across  studies.  Since  this  research  aggregates  results  froin 
independent  studies,  an  element  of  heterogeneity  is  expected  to  be  introduced  through 
differences  in  student/physician  populations,  instructional  approaches  compared, 
outcomes  measured,  and  other  methodological  and  substantive  factors.  As  discussed 
earlier,  some  of  these  differences  are  examined  to  gain  insight  as  to  whether  they 
moderate  the  efficacy  of  problem-based  teaching  and  learning.  An  important  aim  of  this 


122 


study  is  to  test  whether  the  average  effect  size  estimate  for  an  outcome  (i.e.,  main  effect) 
is  representative  of  all  studies  in  a  given  data  set,  or  whether  there  is  statistical  instability 
attributable  to  conditional  variability  (Hedges  &  Olkin,  1985;  Hunter  &  Schmidt,  1994). 

As  previously  discussed,  variations  due  to  sampling  errors  (random  error)  are 
anticipated.  Hunter  and  Schmidt  (1994)  suggest  if  random  error  accounts  for  at  least 
75%  of  the  variance  ("the  75  percent  rule,"  p.  335),  then  other  statistical  artifacts 
probably  account  for  the  balance  of  variance  and  further  analysis  is  unwarranted.  In  such 
a  case,  the  mean  of  effect  size  estimate  (i.e.,  main  effect)  probably  reflects  "true"  effect 
size.  However,  if  effect  size  estimates  vary  between  studies  more  than  expected  by 
random  error,  effect  sizes  are  considered  too  heterogeneous  to  reflect  a  common  effect 
size  (Cohen,  1988;  Hedges,  1994;  Hedges  &  Olkin,  1985;  Hedges  &  Vevea,  1998). 
Therefore,  statistical  instability  is  examined  by  analytically  exploring  the  relationship 
between  study  features  and  effect  size  estimates  using  models  judged  a /rnon  to  be 
theoretically  or  empirically  sound  (Lipsey  &  Wilson,  2001 ). 

When  guided  by  theory  or  empirical  findings,  detecting,  exploring  and  explaining 
nonrandom  variation  in  a  meta-analysis  model  is  usually  performed  with  a  fixed-effects 
model  (Hedges  &  Olkin,  1985;  Lipsey  &  Wilson,  2001).  Fixed-effect  models  assume  the 
existence  of  a  single  effect  of  exposure  to  all  studies,  thus  ignoring  among-study 
variability  (Hedges,  1994;  Hedges  &  Olkin).  Random-effects  method  assumes  that 
studies  are  a  random  sample  from  a  hypothetical  population  and  that  heterogeneity 
between  studies  can  be  represented  by  a  single  variance.  Consequently,  a  random-effects 
method  assigns  more  weight  to  the  results  of  smaller  studies  than  a  fixed-effects  method, 
thus  effect  size  may  be  inflated  (Hedges).  Typically,  the  choice  of  a  fixed  versus  random 
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effects  model  does  not  influence  effect-size  (point  estimate),  but  a  random  effects  model 
tends  to  expand  the  standard  error  of  estimate,  particularly  in  the  presence  of 
heterogeneity  (Hedges). 

Assumptions  guiding  the  selection  of  fixed-effects  or  random-effects  modeling 
are  controversial  (Cook,  Sackett,  &  Spitzer,  1995;  Lao,  loannidis,  &  Schmid,  1998). 
Nevertheless,  given  the  presence  of  heterogeneity,  it  is  widely  accepted  that  a  careful 
investigation  of  potential  sources  of  heterogeneity  is  more  important  than  the  global 
analysis  provided  by  a  random-effects  model  (Thompson,  1993;  Thompson  &  Pocock, 
1991).  Thus,  it  is  generally  agreed  that  a  fixed-effect  model  is  valid  as  a  test  of  overall 
null  hypothesis.  In  addition,  whether  homogeneity  is  present  or  not,  the  aggregate  result 
from  a  fixed-effect  model  is  an  average  measure  of  treatment  effects  in  analyzed  studies 
(Hedges,  1994;  Lipsey  &  Wilson,  2001). 

For  this  research,  levels  of  independent  variables  were  selected  a  priori  and 
guided  by  theory,  thus,  a  fixed-effects  model  was  used  to  examine  conditional  variance  in 
effect  size  estimates  for  a  given  instructional  outcome  (Hedges,  1994;  Hedges  &  OIkin, 
1985;Keppel,  1991;  Lipsey  &  Wilson,  2001).  Experiment  wise  error  (alpha  level)  was 
set  at  .05  for  statistical  significance  to  evaluate  whether  hypothesized  a  priori  study 
features  moderated  instructional  effects  (Hedges). 

Conventional  analysis  of  variance  is  inappropriate  for  testing  the  association 
between  conditional  variance  and  effect  size  estimates  (Hedges  &  Oklin,  1985;  Hedges, 
1994).  Particularly,  since  individual  effect  size  estimates  exhibit  different  variance 
(inversely  proportional  to  sample  size),  error  variance  among  studies  can  differ 
significantly  (Crocker  &  Algina,  1986).  Thus,  to  examine  whether  outcome  specific 
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effect  size  estimates  generated  by  the  results  of  a  set  of  primary  research  reports  varied 
primarily  due  to  sampling  error  or  to  systematic  disparities  among  studies  in  conjunction 
with  sampling  error,  a  hoinogeneity  statistic,  Q,  is  computed  according  to  Cohen  (1988). 
Total  homogeneity  (Q)  is  a  weighted  total  sum  of  squares  about  grand  mean,  which  can 
be  partitioned  into  two  components:  (a)  homogeneity  between  groups  of  studies  (Qb, 
weighted  sum  of  squares  of  group  means  about  grand  mean)  and  (b)  homogeneity  within 
studies  (Qw,  weighted  sum  of  squares  of  individual  effect  size  estimates  about  respective 
group  means).  Thus,  analysis  of  effect  size  variance  is  analogous  to  one-factor  analysis 
of  variance  because  Q  total  =  Q  between-groups  +  Q  within-groups  (Durlak,  1998). 
Consequently,  Qb  and  Qw  yield  direct  omnibus  tests  of  variation  across  groups  in  mean 
effects  (analogous  to  an  omnibus  F  test)  and  variation  within  groups  of  individual  effects, 
respectively  (Hedges). 

Homogeneity  analysis  involves  contrasting  the  variance  exhibited  by  a  set  of 
outcome  specific  effect  size  estimates  with  the  variance  expected  only  if  the  sampling 
error  is  evident  (Cohen,  1988;  Hall  &  Rosenthal,  1991).  Under  null  hypothesis,  Q 
statistic  approximates  a  chi-squared  distribution  with  k  -  1  degrees  of  freedom,  where  k  is 
the  number  of  effect  sizes  and  evaluates  the  distribution  homogeneity  of  outcome  specific 
effect  size  estimates  derived  from  a  set  of  primary  research  reports  (Cohen;  Hedges, 
1994).  An  assumption  of  the  Q  test  is  analyzed  effect  size  estimates  are  retrieved  from  a 
distribution  of  statistically  independent  effect  size  estimates,  which  are  nomially 
distributed  with  a  variance  affected  by  the  data  (Hedges). 

If  Q  is  not  statistically  significant,  testing  for  homogeneity  ceases,  because  effect 
size  was  consistent  across  studies  suggesting  that  the  findings  represent  a  common 
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population  effect  size,  assuming  the  test  for  inomogeneity  is  based  on  a  sufficient  size 
sample  to  afford  a  statistically  powerful  test  (Hall  &  Rosenthal,  1991).  When 
homogeneity  of  a  data  set  is  rejected,  that  is,  a  significant  chi-squared  is  obtained;  further 
exploration  of  the  data  set  is  performed  via  variance  analyses  of  study  features.  A 
significant  Q  suggests  study  features  moderated  the  magnitude  of  effect  sizes.  That  is, 
the  data  set  includes  two  or  more  distinct  populations  warranting  subsequent  steps  of 
fitting  models  to  examine  whether  study  effects  are  operating  to  moderate  variance  of 
effect  sizes.  In  particular,  withiii-group  homogeneity,  Qw,  describes  variation  among 
effect  sizes  within  each  group  (coded  independent  variables).  Thus,  Qw  is  the  sum  of  p 
separate  within-group  homogeneity  statistics,  Qw  =  Qwi  +  Qw2  +  ■  ■  .  Qwp,  where  p  =  the 
number  of  levels  for  independent  variable  (Hedges,  1994). 

Null  hypothesis  for  within-groups  variation  is  tested  by  comparing  the  observed 
value  of  Qw  with  upper  tail  critical  values  of  chi-squared  distribution  with  k  -  p  degrees 
of  freedom.  If  Qw  exceeds  critical  value  at  100(1  -  alpha)  percent  point,  or  95%  level  of 
chi-squared  distribution,  null  hypothesis  of  variance  homogeneity  is  rejected.  A 
significant  Qw  for  a  level  of  study  feature  suggests  grouping  of  effect  size  estimates  is 
heterogeneous.  Thus,  group  homogeneity  statistics  are  useful  indicators  of  relative 
sources  of  within-group  homogeneity  for  multiple  level  study  features. 

Between  group-homogeneity,  Qb,  describes  variation  between  group  cumulative 
effect  sizes  (Hedges,  1994;  Lipsey  &  Wilson,  2001).  Null  hypothesis  of  conditional 
variance  (between-groups)  is  tested  by  comparing  the  observed  value  of  Qu  with  upper 
tail  critical  values  of  chi-squared  distribution  with  p  -  1  degrees  of  freedom.   If  Qb 
exceeds  critical  value  at  the  95%  level,  null  hypothesis  of  homogeneity  is  rejected.  A 
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significant  Qb  indicates  a  study  feature  significantly  affected  tlie  inagnitude  of  effect  size. 
Thus,  statistic  Qb  yields  a  direct  omnibus  test  for  between-group  differences  in  mean 
effects,  but  does  not  indicate  how  study  feature  levels  differ  (Durlak,  1998).  When  Qb 
was  significant  and  there  were  more  than  two  levels  of  a  study  feature,  the  presence  or 
absence  of  overlap  in  95%  confidence  intervals  was  used  to  test  for  significant 
differences  between  levels  at  .05  level  (Hedges). 

Although  quantifying  and  distilling  variance  attributed  to  effect  size  estimates 
associated  with  outcome  measures  and  methodological  features  is  an  important  aspect  of 
this  research,  a  primary  concern  is  discovering  the  relative  importance,  individually  or 
combined,  of  substantive  features  theoretically  or  empirically  believed  to  predict  the 
efficacy  of  problem-based  teaching  and  learning  (Barrows,  1986,  2000;  Engel,  1997; 
Norman  &  Schmidt,  1992;  Savery  &  Duffy,  1995;  Schon,  1987).  Intercorrelations  were 
computed  (SigmaStat,  V.  2.02,  SPSS,  Chicago,  IL)  to  test  the  strength  of  bivariate 
association  between  effect  size  estimates  and  hypothesized  substantive  features  claimed 
to  uniquely  affect  the  efficacy  of  problem-based  teaching  and  learning  (Barrows) 
including  problem-based  instruction  taxonomy  (reiterative  process  or  not;  dummy  coded 
1  and  0),  problem-type  (ill-structured  or  not;  dummy  coded  1  and  0),  origin  of  learning 
issues  (student  generated  or  no;  dummy  coded  1  and  0),  tutor  expertise  (content  expert  or 
not;  dummy  coded  1  and  0)  and  self-directed  learning  (student  centered  or  not;  dummy 
coded  1  and  0)  (Licht,  1998). 

If  the  bivariate  correlation  of  independent  variable  and  outcome  effect  size  was 
significant  at  .10  level,  it  was  used  in  a  multiple-regression  model  (Hedges,  1994;  Licht, 
1998).  Multiple  regression  modeling  provides  an  opportunity  to  estimate  the  relative 
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contribution  of  each  substantive  variable  to  outcome  effect  size.  Discovering  co-linearity 
among  independent  variables  introduces  problems  when  attempting  to  draw  inferences 
about  the  relative  contribution  of  each  predictor  variable  to  the  accuracy  of  the  model. 
However,  if  one  predictor  variable  is  significantly  correlated  with  other  independent 
variables,  but  not  with  outcome  effect  size,  it  was  modeled  as  a  "suppressor  variable" 
(Licht).  A  suppressor  variable  in  regression  model  functions  to  exclude  variance  in  other 
independent  variable(s)  not  shared  with  outcome  effect  size.  That  is,  irrelevant  variance 
is  obviated,  or  "suppressed"  in  the  model  (Licht). 
Threats  to  Reliability 

Data  for  this  research  are  derived  from  varied  instruments  (e.g.,  licensing 
examinations,  locally  constructed  tests,  self-reports)  employed  to  measure  the  effect  of 
intervention  (e.g.,  instructional  technique)  on  a  dependent  variable  (e.g.,  biomedical 
science  achievement,  clinical  science  achievement,  problem-solving  achievement,  self- 
directed  learning,  attitude  toward  learning).  These  instruinents  include  measurement 
error  (random  enor).  As  discussed  earlier,  measurement  error  affects  group  variance,  but 
not  mean  score  (Crocker  &  Algina,  1986).  Classical  measurement  theory  accounts  for 
this  phenomenon.  An  observed  score  is  defined  as  a  true  score  plus  error  score,  and 
observed  score  variance  equals  variance  of  true  score  plus  variance  of  error  scores.  By 
definition,  the  mean  of  error  scores  is  zero.  Thus,  the  mean  of  observed  scores  is  equal  to 
the  mean  of  true  scores.  Consequently,  error  scores  do  not  influence  the  observed  mean 
score,  but  do  influence  observed  score  variance  (Crocker  &  Algina).  Hunter  and  Schmidt 
(1994)  recommend  that  in  meta-analysis,  effect  size  estimates  be  corrected  for 
measurement  error  by  dividing  effect  size  by  the  square  root  of  the  reliability  coefficient 
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of  the  instrument  employed  to  measure  the  dependent  variable.  However,  unless 
reliability  coefficients  are  available  for  an  entire  data  set  for  a  given  outcome,  effect  size 
estimates  should  not  be  corrected  for  reliability  (Hunter  &  Schmidt). 
Threats  to  Validity 

Publication  bias  and  differential  study  quality  of  primary  studies  are  considered  to 
be  the  two  most  serious  validity  problems  that  can  lead  to  biased  estimates  of  effect  size 
(Cooper,  1998;  Matt  &  Cook,  1994).  Publication  bias  refers  to  the  likelihood  that  studies 
with  statistically  significant  or  interesting  results  are  potentially  more  likely  to  be 
submitted,  published  or  published  more  quickly  than  reports  with  insignificant  results 
(Begg,  1994;  Matt  &  Cook).  Thus,  when  only  published  literature  is  included  in  a  meta- 
analysis, results  potentially  lead  to  a  biased  over-estimation  of  effect  size  (Begg).  This 
deficit  of  nonsignificant  published  studies  is  described  as  a  "file-drawer  problem" 
(Rosenthal,  1979,  p.  638).  Related  biases,  including  pipeline  bias  and  language  bias,  also 
can  prejudice  the  results  of  meta-analysis  (Begg).  Pipeline  bias  occurs  when  significant 
results  are  published  faster  than  non-significant  results.  Language  bias  occurs  when 
researchers  whose  native  tongue  is  not  English  are  more  likely  to  publish  their  non- 
significant results  in  non-English  journals,  but  are  more  likely  to  publish  their  significant 
results  in  English.  Therefore,  a  meta-analysis  including  only  reports  in  English  may  be 
grounded  on  a  biased  set  of  research. 

Publication  bias  in  a  meta-analysis  data  set  can  be  assessed  by  examining  a  plot  of 
effect  size  estimates  for  each  study  against  some  measure  of  its  precision,  for  example, 
variance  of  effect  size  estimate  (Begg,  1994).   If  no  publication  bias  is  present,  the 
resulting  plot  should  appear  to  be  funnel  shaped  with  the  spout  directed  upward.  A 
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funnel  configuration  is  expected  because  studies  of  decreasing  sample  size  have 
increasing  large  variation  in  effect  size  estimates  due  to  random  variation  becoming 
increasingly  influential.  However,  the  mean  effect  size  estimate  should  be  similar 
independent  of  sample  size.  That  is,  one  should  be  able  to  draw  a  vertical  line  through 
mean  effect  size  for  data  set,  and  points  should  be  distributed  symmetrically  for  all  values 
of  standard  error  (Begg).  If  chance  of  publication  is  greater  for  larger  sttidies  with 
statistically  significant  results,  some  small  non-significant  studies  may  not  appear  in  the 
literature.  This  will  produce  asymmetry  in  the  plot.  Begg  also  suggests  employing  a 
rank  correlation  test  (Spearman's  rho)  to  examine  the  relationship  between  standardized 
effect  size  and  sample  size  across  studies.  A  rank  correlation  test  is  a  statistical  analogue 
of  a  funnel  plot.  A  significant  correlation  between  effect  size  and  sample  size  indicates 
publication  bias.  Graphical  assessment  for  pooled  data  and  Speannan's  rho  for 
individual  outcomes  were  computed  to  examine  publication  bias. 

Rosenthal  (1979)  suggested  a  solution  to  publication  bias.  A  "fail-safe  N"  is 
calculated  to  estimate  the  number  of  unreported  or  unretrieved  studies  that  would  have  to 
exist  to  negate  a  significant  effect.  This  procedure  assesses  the  "file  drawer"  hypothesis 
as  a  plausible  rival  explanation  and  threat  to  conclusion  validity.  In  addition,  a  "fail-safe 
N"  also  can  be  interpreted  as  conclusion  tolerance  for  future  null  results  (Rosenthal). 
Fail-safe  N  is  the  number  of  additional  studies,  with  mean  effect  size  of  zero,  required  to 
reduce  cumulative  significance  to  an  alpha  level  of  .05.  The  validity  of  this  method  is 
constrained  because  sample  size,  variance,  and  the  potential  for  negative  effect  size  are 
not  considered.  Therefore,  what  threshold  does  a  fail-safe  value  need  to  reach  to  sponsor 
confidence  in  results?  There  are  no  firm  guidelines,  but  Rosenthal  suggests  5n  +  10, 
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where  n  is  the  number  of  studies  included  in  the  meta-analysis,  is  a  reasonable  critical 
value  to  test  a  fail-safe  calculation.  Furthermore,  the  likelihood  that  file  drawers  would 
contain  five  times  more  studies  than  retrieved  for  analysis  seems  remote  (Rosenthal). 

As  discussed  earlier,  study  quality  is  difficult  to  assess  (Wortman,  1994). 
Methods  used  to  confront  study  quality  in  a  meta-analysis  are  similar  to  those  employed 
for  publication  bias,  in  that  there  is  agreement  that  some  assessment  of  quality  should  be 
made,  but  there  is  minimal  agreement  on  the  procedures  necessary  to  assess  quality,  or 
ways  to  incorporate  results  into  a  meta-analysis  (Begg,  1994;  Glass,  1976;  Wortman). 
For  Glass  "It  is  an  empirical  quesfion  whether  relatively  poorly  designed  studies  give 
results  significantly  at  variance  with  those  of  the  best  designed  studies"  (p.  4).  From  this 
vantage,  methodological  features  are  coded  as  previously  described  and  analyzed  as 
potenfial  modifiers  of  effect  size  estimates.  Diversity  of  methodological  quality  was 
anticipated,  thus  results  were  examined  to  discover  the  extent  to  which  effect  size 
estimates  varied  as  a  function  of  methodological  study  features  (e.g.,  random  assignment, 
baseline  equivalence).  Ultimately,  confidence  in  results  is  warranted  when  similar  results 
emerge  independent  of  methodological  features  (Lipsey  &  Wilson,  2001). 
Conceptual  Summary  of  Methodology 
Standardized  effect  size  estimates  were  computed  for  all  dependent  variables  in 
each  study  for  five  categories  (main  effects)  of  outcome  measures  (biomedical  science 
achievement,  clinical  science  achievement,  problem-solving  achievement,  self-directed 
learning  skills,  and  attitude  toward  learning).  Effect  size  quanfifies  the  average 
performance  of  problem-based  instruction  groups  in  relation  to  lecture-based  instruction 
groups.  For  example,  an  effect  size  of  1 .0  indicates  that  a  problem-based  group 


131 


perfonned  one  standard  deviation  above  a  lecture-based  group.  Effect  sizes  also  can  be 
operationalized  as  the  average  percentile  standing  of  an  average  treated  participant 
(problem-based  learning)  relative  to  an  average  untreated  participant  (lecture-based 
learning).  For  example,  ES  =  1 .0  transformed  into  percentile  ranks  (assuming  a  normal 
distribution  of  data),  the  lecture-based  group  performed  at  50"'  percentile  and  the 
problem-based  group  performed  at  84'*'  percentile.  If  an  effect  size  is  negative,  it 
indicates  that  the  lecture-based  group  out-performed  the  problem-based  group.  Similarly, 
an  effective  size  of -1.0  reflects  lecture-based  group  performance  exceeded  problem- 
based  group  performance  by  one  standard  deviation. 

Theoretically  and  empirically  grounded  features  (outcome  measures, 
methodological  procedures,  and  substantive  characteristics)  were  coded  for  each  study. 
Grouped  effect  size  estimates  were  plotted  and  outliers  removed  for  differential  analyses. 
Sensitivity  analysis  was  performed  to  assay  effect  magnitude  of  excluding  outlier  data. 
Each  effect  size  estimate  was  weighted  using  a  reciprocal  of  its  sample  variance  (Lipsey 
&  Wilson,  2001).  Fixed-effect  analysis  of  variance  was  employed  to  evaluate  whether 
aggregated  effect  size  estimates  shared  a  common  effect  size.  A  homogeneity  statistic, 
Qt,  was  computed  for  each  main  effect.  Significant  Q  at  the  .05  level  rejects  the  null 
hypothesis  of  homogeneity  indicating  that  the  variance  is  greater  than  expected  due  to 
sampling  error,  thus,  suggesting  fixed-effects  variance  analyses  to  assess  conditional 
moderation  of  effect  size  magnitude  by  coded  study  features.   When  a  moderating  feature 
found  significant  included  three  or  more  levels,  95%  confidence  intervals  were  examined 
to  distinguish  levels  of  the  study  feature  as  significant  moderators  of  effect  size  estimates 
at  .05  level.  Intercorrelations  among  theoretically  and  empirically  grounded  dimensions 
of  problem-based  instruction  and  outcome  effect  size  estimates  were  computed. 
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Substantive  variables  correlated  significantly  at  the  .  10  level  or  less,  were  entered  into  a 

multiple-regression  analysis  to  examine  their  relative  contribution  to  outcome  effect  size. 

Critique  of  Methodology 

Glass  (1976)  posited  four  potential  limitations  of  meta-analysis  including 

selection  bias,  methodologic  quality,  nonindependcnce,  and  commensurability.  Selection 

bias  or  publication  bias  may  confound  meta-analysis.  An  assumption  of  meta-analysis  is 

that  eligible  available  research  studies  are  viewed  as  a  population  to  be  systematically 

sampled.  Although  some  investigators  argue  against  using  unpublished  data  because  the 

data  are  not  peer  reviewed,  unpublished  research  (e.g.,  dissertations,  conference  reports) 

potentially  represent  a  significant  amount  of  what  is  known  (Dickersin  &  Berlin,  1992). 

Glass  favored  inclusion  in  meta-analysis  of  all  published  and  unpublished  reports  despite 

methodological  quality.  Specifically,  Glass  argued  that  if  design  flaws  are  important, 

there  would  be  a  correladon  between  study  quality  and  effect  size.  Variance  analyses  in 

the  current  study  indirectly  confirmed  this  observation.  Moreover,  Glass  (1977)  argues 

"there  are  no  perfect  studies,  but  it  is  reasonable  to  believe  that  together  imperfect  studies 

can  converge  on  a  legifimate  conclusion"  (p.  356).  Contrary  to  Glass'  position, 

Mansfield  and  Busse  (1977)  argue  only  research  void  of  methodological  flaws  should  be 

selected  for  meta-analysis.  They  believe  an  accurate  estimate  of  treatment  strength  can 

be  derived  only  from  studies  incorporating  strong  methodological  design.  However, 

Campbell  and  Stanley  (1963)  endorse  a  comprehensive  inclusion  approach,  as  they  note: 

The  more  numerous  and  independent  the  ways  in  which  the  experimental  effect  is 
demonstrated,  the  less  numerous  and  less  plausible  any  singular  rival  invalidating 
hypothesis  becoines.  The  appeal  is  to  parsimony.  The  "validity"  of  the 
experiment  becomes  one  of  the  relative  credibility  of  rival  theories:  the  theory  that 
X  had  an  effect  versus  the  theories  of  causation  involving  the  uncontrolled 
factors.  If  several  sets  separate  uncontrolled-variable  effects  must  be 
hypothesized,  a  different  one  for  each  observed  difference,  then  the  effect  of  X 
becomes  the  most  tenable,  (p.  217) 
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Moreover,  this  comprehensive  approach  makes  possible  the  critical  assessment  of  a 
research  question  for  a  given  topic.  Thus,  considering  all  available  primary  research 
comparing  problem-based  and  lecture-based  instruction  independent  of  methodological 
quality  is  important  because  meta-analysis  can  reveal  whether  methodological, 
contextual,  or  substantive  variations  in  primary  research  studies  are  associated  with 
variation  in  effect  sizes.  Eventually  ruling  out  methodological  factors  as  plausible  rival 
explanations  is  essential  before  drawing  substantive  conclusions  in  any  research  endeavor 
(Campbell  &  Stanley,  1966). 

Some  studies  have  multiple  treatments  and  report  two  or  more  comparisons  for 
indicated  independent  variable.  In  the  later  case,  Lipsey  and  Wilson  (2001 )  recommend 
treating  each  effect  size  as  independent.  Glass  (1996)  argues  aggregating  studies  is 
similar  to  grouping  subjects  of  varied  characteristics  and  abilities,  for  example,  individual 
differences  of  participants.  Hedges  and  Olkin,  (1985)  suggest  effect  size  is  most  useful 
when  computed  from  measures  that  represent  the  same  construct.  Wortman  (1994) 
suggests  the  study  should  be  the  unit  of  analysis.  According  to  Wortman,  if  a  study  has 
multiple  effect  sizes,  individual  effect  sizes  should  be  averaged  for  a  cumulative  study 
effect. 

Meta-analysis  has  been  criticized  for  commensurability,  or  an  "apples  &  oranges" 
problem,  which  suggests  different  phenomena  are  aggregated  inappropriately  (Glass, 
1 976;  Gleser  &  Olkin,  1 994;  Wortman,  1 994).   In  response.  Hunter  and  Schmidt  ( 1 994) 
argue  grouping  dependent  variables  into  categories  that  are  grounded  in  theory  or 
logically  coherent  is  appropriate.  For  example,  apples  and  oranges  are  fruits  (Glass). 
Indeed,  a  convergence,  or  "triangulation,"  of  findings  from  methodologically  varying 
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investigations  strengthens  the  vahdity  of  an  effect  (Campbell  &  Fisk,  1959).  For 
Campbell  and  Fisk,  when  a  relationship  is  reproduced  with  varied  circumstances,  it  is 
robust.  Thus,  to  that  extent,  the  main  effects  of  this  research  are  consistent  (i.e., 
statistically  similar)  despite  the  varied  research  methods  and  the  varied  methods  of 
implementing  of  problem-based  and  lecture  based  teaching  and  learning;  confidence  in 
the  results  is  warranted. 


CHAPTER  4 
RESULTS  AND  ANALYSES 

Studies  included  in  the  present  research  were  reported  between  November  1 977 
and  December  2002,  and  involved  forty- five  universities  in  twelve  countries  and  12,979 
participants  (PBL  =  4,227;  LBL  =  8,752).  An  inclusive  approach  was  adopted;  no 
decisions  were  made  regarding  the  quality  of  primary  research  reported.  This  was  an 
attempt  to  maximize  comparative  evidence  and  to  examine  a  full  range  of  instructional 
and  research  practices  comparing  the  relative  merits  of  lecture-based  versus  problem- 
based  teaching  and  learning  in  medical  students  and  physicians.  Of  1 16  studies  retrieved, 
eighty-two  individual  reports  provided  data  consistent  with  computing  standardized 
effect  size  estimates.  I  queried  authors  of  one  study  (Enarson  &  Cariaga-Lo,  2001 )  who 
reported  unusual  variance  for  licensing  examination  results  and  received  corrected  data 
via  Internet  mail.  Studies  included  in  this  research  are  listed  in  Appendix  A.  Thirty-four 
retrieved  studies  were  not  analyzed  because  the  data  to  compute  standardized  effect  size 
estimates  were  not  reported  and  not  obtainable  from  the  authors.  Excluded  studies  and 
the  reasons  for  their  exclusion  are  listed  in  Appendix  B.  Box-plots  of  standardized  effect 
size  estimates  revealed  outliers  for  biomedical  science  achievement  (n  =  1 ),  clmical 
science  achievement  (n  =  I )  and  attitude  toward  learning  (n  =  1 )  outcomes.  Outliers  did 
not  eliminate  entire  studies  because  other  outcomes  reported  were  retained  for  analyses. 
Included  studies  presented  findings  for  biomedical  science  achievement  (n  =  33),  clinical 
science  achievement  (n  =  29),  problem  solving  achievement  (n  =  10),  self-directed 
learning  skills  (n  =  19),  and  attitude  toward  learning  (n  =  30).  One  report 
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(Michel,Bisclioff,  zii  Heringdorf,  Neumann,  &  Jakobs,  2002)  presented  data  from  two 
sampling  designs,  one  randomized  and  one  self-selected.  Since  distinct  sampling  designs 
were  employed  to  examine  instructional  effect  for  biomedical  science  achievement,  self- 
directed  learning,  and  attitude  toward  learning,  these  outcome  data  were  analyzed  as 
independent  studies.  In  accordance  with  Hunter  and  Schmidt  (1994),  effect  size 
estimates  were  not  corrected  for  reliability  because  few  outcome  measures  were 
accompanied  with  reliability  coefficients. 

Distribution  of  Effect  Sizes 
Effect  sizes  using  report  as  the  unit  of  analysis  (N  =  82)  are  plotted  against  variance 
(Figure  4-1)  and  there  is  no  significant  relationship  (r  =  .01,  p  =  .91 ).  Additional  support 
refuting  publication  bias  is  sponsored  because  the  hypothesis  stipulating  effect  sizes  are 
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Figure  4-1.  Effect  size  estimates  for  82  reports  are  plotted  as  a  function  of  associated 
variance.  Note:  Line  correlating  effect  size  and  variance  is  plotted. 
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normally  distributed  cannot  be  rejected  according  to  Kolmogorov-  Smimov's  test,  D  = 
.10,  E  =  .20  (Figure  4-2).  As  discussed  in  Chapter  3,  Spearman's  rho  is  reported  as  an 
index  of  publication  bias  for  individual  outcome  effect  sizes  (Begg,  1994). 
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Figure  4-2.  Frequency  distribution  of  problem-based  learning  effect  sizes  for  121 

outcome  measures.  Note.  Expected  nomial  distribution  curve  is  plotted. 

Main  Effects  of  Problem-Based  Learning 

Weighted  and  unweighted  main  effect  size  estimates  and  associated  variance 
analyses  are  summarized  in  Table  4-1.  Failure  to  weight  effect  sizes  appropriately  may 
yield  biased  average  effect  estimates  and  incorrect  sampling  errors  if  effect  sizes  from 
different  investigations  vary  in  precision.  As  discussed  in  Chapter  3,  effect  size  estimates 
were  weighted  using  inverse  of  variance,  affording  greater  weight  to  more  precise 
estimates  (Lipsey  &  Wilson,  2001).  Variance  of  effect  size  estimates  for  biomedical 
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science  achievement  (Qt  =  1 33 .6,  df  =  32,  e  <  -0000 1 ).  clinical  science  acliievement  (Q  r 
=  162.8,  df  =  28,  E  <  .00001 ),  self-directed  learning  skills  (Q,  =1 1 1.8,  df  =  18,  e  <  .001), 
and  attitude  toward  learning  (Q,  =138.9,  df  =  29,  2  <  .00001)  are  significant  and  cannot 
be  explained  entirely  by  sampling  error.  Therefore,  as  indicated  in  Chapter  3,  sub- 
grouping  according  to  a  priori  study  features  is  indicated  to  discover  potential  moderators 
of  differential  instructional  effects.  Variance  of  effect  size  estimates  for  problem-solving 
achievement  was  homogeneous  (Qj  =  8.1,  df=  9,  p  =  .53).  Consequently,  the  null 
hypothesis  is  not  rejected  and  this  sampling  error  alone  explains  the  magnitude  of 
variance  in  problem-solving  effect  sizes.  Accordingly,  the  mean  of  problem-solving 
effect  size  estimates  is  assumed  to  reflect  "true"  effect  size. 
Table  4-1.  Main  effects  of  problem-based  versus  lecture-based  learning 


Mean  Effect  Size 95%CI  Homogeneity" 

Outcome N  Unweighted    Weighted    Lower    Upper (Q) 

Biomedical  achievement    33  -.01 

Clinical  achievement          29  .31 

Problem-solving                 10  .30 

Self-directed  learning         19  .54 

Attitude  toward  learning    30  .52 


.02 

-.02 

.06 

133.6** 

.32 

.27 

.38 

162.8** 

.19 

.03 

.34 

8.1 

.47 

.39 

.55 

111.8** 

.45 

.39 

.52 

138.9** 

Note.  N  =  number  of  independent  outcome  measures  of  effect  size  (ES).  If  95% 
confidence  interval  (CI)  for  weighted  effect  size  does  not  contain  zero,  ES  is  significant 
at  .05  level.  "Significance  indicates  rejection  of  hypothesis  of  homogeneity. 
**E<.01. 

Detailed  results  and  analyses  are  organized  according  to  hypotheses  associated 

with  particular  research  questions.  Representative  examples  of  statistically  analyzed 

research  also  are  examined  via  narrative  to  illustrate  research  design,  methods  and 

statistical  conclusions.  Narrative  analysis  also  is  provided  of  outlier  data  and  studies  not 

amenable  to  computation  of  standardized  effect  size,  but  still  of  sufficient  methodological 

quality  to  provide  insight  regarding  important  differential  cognitive  and  non-cognitive 

effects  of  problem-based  versus  lectured-based  insti-uction. 
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Biomedical  Science  Achievement 
Hypothesis  1.  Aggregated  effect  sizes  for  biomedical  science  achievement  in 
students  and  physicians  wlio  undergo  medical  education  with  problem-based  compared  to 
lecture-based  teaching  and  learning  will  be  significantly  positive  because  the  lower  limit 
of  the  95%  confidence  interval  will  exceed  zero.  Biomedical  science  achievement 
measures  included  scores  from  standardized  examinations  and  locally  developed  tests, 
primarily  incorporating  multiple-choice  questions  or  multi-method  assessments. 
Standardized  effect  size  estimates  and  associated  95%  confidence  intervals  were 
computed  using  biomedical  science  achievement  outcomes  presented  in  thirty-three 
studies  involving  12,298  subjects  are  summarized  graphically  in  Figure  4-3.  Exploring 
the  plot  of  95%  confidence  intervals  for  each  effect  size  reveals  five  confidence  intervals 
are  above  zero,  three  are  below  zero,  and  twenty-five  include  zero.  The  mean  effect  size 
is  included  in  25  of  33  confidence  intervals.  The  findings  suggest  no  significant 
difference  in  biomedical  science  knowledge  gain  by  students  and  physicians  undergoing 
problem-based  (n  =  3,815)  versus  lectured-based  teaching  and  learning  (n  =  8,483).  ES 
=0.02,  95%CI  -0.02  to  0.06.  However,  power  of  this  analysis  is  only  0.27. 
Consequently,  sustaining  the  null  hypothesis  should  be  viewed  with  caution  as  the 
probability  of  a  Type  II  error  is  73%.  What  was  the  effect  of  excluding  outlier  (ES  = 
0.88,  n  =  370)?  Sensitivity  analysis  revealed  computing  with  outlier,  ES  =  0.04,  95%CI 
0.001  to  0.083,  does  not  differ  significantly  or  materially  from  mean  effect  size  estimate 
and  the  confidence  interval  derived  with  a  reduced  data  set.  Publication  bias  for 
biomedical  science  achievement  outcomes  appears  unlikely  because  there  was  no 
significant  correlation  between  effect  size  estimates  and  sample  size  (Spearman's  rho  = 
.12,2=. 51). 
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Figure  4-3.  Plot  of  standardized  effect  sizes  and  associated  95%  confidence  intervals  for 
biomedical  science  achievement.  Note.  Data  are  derived  from  33 
independent  studies  involving  12,298  participants.  Study  number  87  appears 
twice  because  two  independent  samples  were  reported.  Citation  for  study 
number  is  listed  in  Appendix  A.  Dashed  line  represents  mean  weighted 
effect  size. 

Hypothesis  2.  Variation  in  biomedical  science  achievement  effect  sizes  will 

exceed  random  error  as  evidenced  by  a  Q-statistic  at  the  .05  level  of  significance 

warranting  sub-group  analyses  to  test  for  potential  moderation  by  a  priori  defined 

outcome  measures,  methodological  features,  and  substantive  features  of  problem-based 

teaching  and  learning.  Frequency  distributions  of  standardized  effect  sizes  are 

represented  in  Figure  4-4.  Variance  analysis  suggests  the  distribution  of  effect  sizes  for 

biomedical  science  achievement  is  greater  than  would  be  expected  entirely  from 

sampling  error,  Qt(32)  =  133.6,  p  <  .00001 .  Consequently,  homogeneity  hypothesis  is 

rejected  implying  that  there  are  systematic  differences  among  studies  in  addition  to 
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sampling  error,  thus  further  exploration  is  warranted  to  discover  whether  study  features 
are  operating  to  moderate  variance  of  effect  sizes.  The  distribution  of  within-group  and 
between-group  variances  of  effect  size  estimates  associated  with  outcome, 
methodological,  and  substantive  features  are  summarized  in  Tables  4-2,  4-3  and  4-4, 
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Figure  4-4.  Frequency  distribution  of  effect  sizes  for  biomedical  science  achievement. 
Expected  normal  distribution  curve  is  illustrated.  Note.  Data  are  derived 
from  33  independent  studies  involving  12,298  participants. 

Outcome  measurement  features.  Biomedical  science  achievement  effect  sizes 

significantly  varied  with  assessment  instrument  Qb(1  )  =  9.9,  p  =  .002,  and  are 

summarized  in  Table  4-2.  Achievement  gain  measured  with  licensing  examinations 

slightly  favored  those  who  underwent  lecture-based  versus  problem-based  instruction,  ES 

=  -0.06,  but  the  difference  was  not  significant,  95%CI  -0.13  to  0.01.  Locally  constructed 

tests  (mostly  criterion-referenced  assessments)  significantly  favored  students  undergoing 
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problem-based  instruction  (ES  =  0.07.  95%CI  0.02  to  0.13),  however,  significant 

variability  persisted  within  individual  groupings:  licensing  tests,  Qw,  =  88.5,  g<  .001, 

and  local  tests,  Qw,  =  35.2,  g  =  .002.  Therefore,  groups  were  stratified  according  to 

outcome  type.  Type  of  licensing  examination  significantly  moderated  the  effect  size 

estimate,  Qn(l)  =  6.3,  p  =  .01,  however,  92%  of  Qw  resided  in  data  derived  from  NBME 

Part  I  examination  scores.  There  were  no  significant  differential  effects  of  instruction 

when  studies  assessed  biomedical  science  achievement  using  NBME  Part  1  (ES  =  -0.02) 

or  USMLE  Step  1  (ES  =  -0.22).  The  latter  finding  is  suiprising,  because  in  1992, 

USMLE  Step  I  replaced  NBME  Part  I  in  an  attempt  to  de-emphasize  factual  recall  in 

favor  of  items  facilitating  expression  of  higher-order  thinking  skills  (O'Donnell,  et  al., 

1993).  This  shift  in  the  orientation  of  licensing  assessment  is  more  consistent  with  the 

Table  4-2.  Mean  effect  sizes  and  variance  distribution  for  biomedical  science 
achievement  as  a  function  of  outcome  measure 


N 

n 

ES 

95%CI 

Homogeneity 

Variables 

Lower 

Upper 

On" 

Qw," 

Outcome  source 

9.9** 

Licensing  tests 

17 

4,877 

-.06 

-.13 

.01 

88.5** 

Local  tests 

16 

7,385 

.07 

.02 

.13 

35.2** 

Outcome  type 

Licensing  tests 

6.3* 

NBME  Part  1 

14 

3,620 

-.02 

-.10 

.07 

75.4** 

USMLE  Step  1 

3 

1,257 

-  "11 

-.52 

.09 

6.9* 

Local  tests 

1.3 

Multimethod  tests 

5 

1,301 

.13 

-.03 

.30 

4.1 

MCQ  tests 

11 

6,084 

.06 

-.01 

.12 

29.8* 

Note.  N  =  number  of  weighted  effect  size  (ES)  estimates  averaged;  n  =  number  of 
subjects.  If  95%  confidence  interval  (CI)  does  not  contain  zero,  ES  is  significant  at  .05 
level.  Qb  =  between-group  variance;  Qwi  =  within-group  group  variance  for  each  feature 
sub-group;  NBME  =  National  Board  of  Medical  Examiners  Part  I;  USMLE  =  United 
States  Medical  Licensing  Examination  Step  I,  which  replaced  NBME  Part  I  in  1992; 
multimethod  tests  =  varied  assessment  strategies  including  multiple  choice  questions 
(MCQ),  short  answer,  essay,  and  matching  items.  "Significance  indicates  rejection  of 
hypothesis  of  between-group  homogeneity.  "Significance  indicates  rejection  of 
hypothesis  of  within-group  homogeneity. 
*E<.05,  **2<.0I. 


143 

aims  of  problem-based  than  lecture-based  teaching  and  learning,  thus,  theoretically,  PBL 
students  should  be  favored,  however,  only  three  included  studies  compared  achievement 
outcome  with  USMLE  Step  1  data.  As  discussed  later,  one  outlier  study  compared 
instructional  affect  with  USMLE  Step  1  performance. 

Interestingly,  the  central  tendency  of  effect  sizes  for  problein-based  instruction 
was  significantly  positive  when  assessed  with  locally  constructed  tests,  and  within-group 
variance  was  signiHcant  (Qw  =  35.2,  g  =  .002).  When  further  stratified,  effect  size  was 
positive  when  assessed  with  locally  constructed  multi-method  (ES  =  0.13)  and  multiple- 
choice  question  (ES  =  0.06)  assessments,  but  the  effects  were  not  significant  nor  was 
there  a  difference  in  intergroup  variance  (Qb  =  1.3,  p  =  .25). 

Substantial  heterogeneity  resides  in  effect  sizes  estimated  with  licensing 
examination  (Qw,  =  88.5);  this  is  diminished  only  slightly  after  stratifying  according  to 
outcome  type.  In  contrast  effect  sizes  estimated  with  non-licensing  assessments  represent 
much  less  heterogeneity  (34%  of  Q,).  Non-licensing  assessments  were  employed  in  16 
independent  studies  including  7,385  subjects  (PBL  =  2,369;  LBL  =  5,016).  The  mean 
effect  size  from  this  data  set  is  0.07,  which  is  significant,  because  the  95%CI  0.02  to  0.13 
does  not  include  zero.  There  is  no  evidence  that  publication  bias  is  operating  for  this 
reduced  sample  (Spearman's  rho  =  .16,  p  =  .56).  Since  locally  constructed  assessments 
likely  are  calibrated  to  instructional  objectives  independent  of  instructional  method 
(Biggs,  1996),  this  reduced  data  set  was  employed  to  examine  potential  methodological 
and  substantive  moderators  of  effect  size  for  biomedical  science  achievement.  Findings 
from  this  reduced  data  set,  however,  should  be  interpreted  with  caution  because  outcome 
measures  closely  parallel  the  objectives  of  problem-based  rather  than  lecture-based 
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instruction.  Tlius,  the  external  validity  of  this  reduced  data  set  may  be  jeopardized  by  the 

reactive  or  interaction  effect  of  assessment  (Campbell  &  Stanley,  1966). 

Methodological  features.  Results  of  the  analyses  are  presented  in  Table  4-3. 

Whether  or  not  reliability  of  outcome  measure  was  considered  in  the  data,  analysis  for 

biomedical  science  achievement  is  the  only  method-related  moderator  of  effect  size. 

Table  4-3.  Mean  effect  sizes  and  variance  analyses  for  biomedical  science  achievement 
as  a  function  of  methodological  features 


N 

ES 

95%CI 

Homogeneity 

Variables 

Lower 

Upper 

Qb" 

Qw,' 

Sampling  method 

1.4 

Randoin  assignment 

5 

.16 

-.06 

.37 

2.6 

Intact  groups 

11 

.06 

.00 

.12 

31.2 

Baseline  equivalence 

0.1 

Yes 

12 

.07 

.01 

.13 

33.6 

No 

0 

Not  reported 

4 

.05 

-.22 

.33 

1.6 

Control  group 

9.7 

Parallel  curriculum  /  course 

9 

.08 

-.05 

.22 

12.5 

Another  institution 

3 

.09 

-.04 

.22 

1.3 

Historical  curriculum  /  course 

4 

-.18 

-.45 

.09 

11.7 

Longitudinal  design 

0.1 

Yes 

3 

.07 

-.07 

.21 

17.8 

No 

13 

.08 

-.02 

.17 

17.4 

Attrition  considered 

0.9 

Yes 

3 

.14 

-.22 

.50 

0.2 

No 

13 

.06 

-.01 

.02 

30.3 

Reliability  considered 

16.8* 

Yes 

3 

-.28 

-.67 

.11 

7.2 

No 

13 

.10 

.05 

.16 

11.2 

Validity  considered 

1.3 

Yes 

8 

.09 

.02 

.16 

9.5 

No 

8 

.02 

-.11 

.15 

24.4 

Publication  year 

2.6 

1980-1989 

2 

-.06 

-.21 

.11 

2.3 

1990-1999 

10 

.07 

.01 

.13 

28.5 

2000-2002 

4 

.09 

-.16 

.34 

0.1 

Note.  N  =  number  of  weighted  effect  size  (ES)  estimates  averaged.  If  95%  confidence 
interval  (CI)  does  not  contain  zero,  ES  is  significant  at  .05  level.  Qb  =  between-group 
variance;  Qw,  =  within-group  variance  for  each  feature  sub-group.  ^Significance 
indicates  rejection  of  hypothesis  of  between-group  homogeneity.  ""Significance  indicates 
rejection  of  hypothesis  of  within-group  homogeneity. 
*E  <  .05 
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Studies  not  including  reliability  of  outcome  measure  in  statistical  analyses 
exhibited  a  significant  positive  effect  size,  ES  =  0.10,  95%CI  0.05  to  0.16. 
Unfortunately,  only  21%  of  total  variance  resides  in  studies  incorporating  the 
demonstrated  reliability  of  achievement  measures.  Baseline  equivalence  was  addressed 
in  75%  of  studies  and  effect  size  was  slightly,  but  significantly  positive.  Although  effect 
sizes  derived  from  a  historical  control  were  employed  in  a  few  studies  (n  =  4),  they 
represented  52%  of  Qw  and  produced  a  mean  effect  size  (ES  =  -0. 1 8)  outside  95%CI  for 
coincident  cohorts  at  the  same  (ES  =  0.08)  or  another  (ES  =  0.09)  institution.  Attntion 
was  considered  in  only  19%  of  studies.  When  a  validated  outcome  measure  was  used,  ES 
was  significamly  positive  (ES  =  0.09,  95%CI  .02  to  .16),  but  not  statistically 
distinguishable  from  studies  not  using  validated  measures  of  effect  size.  Studies  reported 
from  1990  through  1999  contributed  81%  of  total  variance  and  averaged  a  small,  but 
significant  effect  size,  ES  =  0.07,  95%CI  .01  to  .13. 

Substantive  features.  Variance  analyses  of  substantive  features  are  summarized 
in  Table  4-4.  The  magnitude  of  effect  sizes  did  not  differ  as  a  function  of  studies  with 
varied  scopes  of  problem-based  instruction,  Qb(2)  =  0.1,  p  =  .98;  although  54%  of 
within-group  variance  was  located  in  studies  conducted  with  curriculum-wide 
implementation  ofthe  instructional  method.  Student  experience,  Qn(2)  =  8.1,  2=  .01, 
with  problem-based  instruction  moderated  effect  sizes,  but  no  experience  level  was 
associated  with  significant  effect,  and  60%  of  Qw  was  located  in  studies  involving 
participants  with  no  previous  PBL  experience.  Reiterative  problem-based  instruction 
produced  a  significant  effect  size  (ES  =  0.07,  95%CI  .01  to  .14),  but  the  difference  with 
more  general  implementation  of  PBL  was  not  significant,  Qb(1)  =  0.0,  g  =  .99. 
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Table  4-4.  Mean  effect  sizes  and  variance  analyses  for  biomedical  science  achievement 
as  a  function  of  substantive  features 


95%CI 


Variables 


ES 


Homogeneity 


Scope  of  instruction 

Curriculum 

Entire  course 

Part  of  a  course 
Students'  PBL  experience 

0  years 

1  year 

2  or  more  years 
PBL  taxonomy 

Reiterative  PBL 

Problem-based 
Problem  type 

Ill-structured 

Mixed  structure 
Learning  issues  responsibility 

Student  generated 

Faculty-student  generated 
Tutor  expertise 

Content  expert 

Non-content  expert 

Peer  tutor 
Self-directed  learning 

Student  centered 

Plus  mandatory  lectures 

Plus  elective  lectures 


Lower       Upper        Qb" 


0.1 


5 

.07 

-0.02 

.15 

8 

.09 

-.06 

.24 

3 

.07 

-.26 

.40 

12 

.09 

-.02 

.19 

2 

-.26 

-.63 

.11 

2 

.09 

-.09 

.10 

9 

.07 

.01 

.14 

6 

.08 

-.09 

.24 

4 

.09 

-.01 

.18 

7 

-.13 

-.29 

.04 

5 

.13 

-.06 

.31 

10 

.06 

-.01 

.12 

12 

.07 

.01 

.13 

2 

.22 

-.89 

1.3 

1 

-.02 

-.44 

.40 

7 

.08 

.01 

.15 

6 

.13 

-.12 

.39 

3 

-.25 

-.71 

.21 

0.0 


0.9 


2.8 


11.3* 


19.0 

7.5 

8.7 

16.2 
9.6 

1.3 

26.2 


5.0 
13.8 

11.7 
22.5 

26.2 
0.4 
0.0 

13.2 
2.3 
8.4 


Note.  N  =  number  of  weighted  effect  size  (ES)  estimates  averaged.  If  95%  confidence 
interval  (CI)  does  not  contain  zero,  ES  is  significant  at  .05  level.  Qb  =  between-group 
variance;  Qwi  =  within-group  variance  for  each  feature  sub-group.  N  varies  among 
independent  variables  because  some  features  could  not  be  ascertained.  'Significance 
indicates  rejection  of  hypothesis  of  between-group  homogeneity. 
*p<.05.  **E<.01. 

Problem-type  moderated  effect  size  estimates,  Qb(1)  =  8.8,  p  =  .01,  with  ill-structured 

problems  (ES  =  0.09)  and  mixed  (combined  ill-structured  and  structured)  problems  (ES  = 

-0.13)  producing  opposite  but  statistically  equivalent  effects.   Effect  size  was  not 

influenced  by  origin  of  learning  issues,  Qb(  1 )  =  0.9,  p  =  .35.  Tutor  expertise  was  not  a 

significant  predictor  of  effect  size  estimates,  Qb(1)  =  2.8,  g  =  .09.  Distribution  of  studies 

and  variance  favored  tutors  with  content  expertise  (72%  of  Qw).  The  characteristics  of 
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self-directed  learning  modulated  effect  size  estimates,  Qb(2)  =  1 1 .3,  e  =  .004.  When 
instructional  contrasts  involved  self-directed  learning  guided  by  students  without  formal 
or  informal  extra-tutorial  group  influences,  mean  effect  size  was  positive  and  significant, 
ES  =  0.08,  95%C1  0.01  to  0.15. 

Table  4-5  presents  the  correlation  matrix  of  hypothesized  positive  predicative 
dimensions  for  PBL  efficacy  as  discussed  in  Chapter  2.  There  were  no  bivariate 
correlations  between  substantive  features  and  mean  effect  size  estimates  at  the 
significance  level  of  .10  or  less.  Thus,  multiple  regression  analysis  is  not  indicated. 
Table  4-5.  Intercorrelations  in  dimensions  of  PBL  for  biomedical  science  achievement 


Variable A B C D E 

Mean  effect  size  estimate  -.05  .08  .42  -.  1 7  .23 

A.  PBL  taxonomy  .60*  .00  .61*  .24 

B.  Problem  type  -.41  .36  .24 
C  Learning  issues  responsibility                                                                 -.19          .5g* 

D.  Tutor  expertise  .20 

E.  Self-directed  learning 

Note.  Data  are  derived  from  the  reduced  data  set  of  16  studies.  PBL  taxonomy  = 
reiterative  or  not;  Problem  type  =  ill-structured  or  not;  learning  issues  responsibility  = 
student  derived  or  not;  Tutor  expertise  =  content  expert  or  not;  Self-directed  learning  = 
student  centered  or  not. 
*E<.10. 

There  were  significant  intercorrelations  between  PBL  taxonomy  and  problem  type 

(r  =  .60,  p  =  .04)  and  tutor  expertise  (r  =  .61 ,  p  =  .02)  as  well  as  responsibility  for  learning 

issues  and  self-directed  learning  (r  =  .58,  p  =  .02). 

Examination  of  Outlier  Study 

Blake  and  associates  (2000)  compared  USMLE  Step  1  scores  in  an  initial  four 

consecufive  classes  of  students  completing  preclinical  (biomedical  sciences)  training  with 

PBL  and  two  last  classes  who  underwent  LBL  at  the  University  of  Missouri  -  Columbia. 

Pre-clinical  learning  for  LBL  students  was  driven  by  "discipline-based,  departmentally 

conducted  courses  that  relied  on  lectures  and  laboratory  experiences"  (p.  66).  Indeed,  "in 
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1988,  a  member  of  a  visiting  Liaison  Committee  on  Medical  Education  accreditation  had 
described  the  currictiUmi  as  perfectly  preserved  from  the  1960s"  (p.  66).  After 
conversion  to  a  problem-based  curriculum,  preclinical  learning  was  facilitated  by  faculty 
tutored  groups  of  eight  students  convening  for  collaboration  approximately  ten  hours 
weekly.  Discussions  were  not  discipline  bound.  Tutorials  were  supplemented  weekly 
with  not  more  than  ten  hours  of  required  traditional  lectures  and  laboratory.  Investigators 
claim  a  baseline  equivalence  between  357  PBL  and  240  LBL  students,  who  served  as 
historical  control,  because  academic  characteristics  of  matriculates  were  similar  before 
and  after  the  curricular  change. 

Mean  USMLE  Step  1  scores  for  PBL  students  m  1997  (M  =  207,  SD  =  18),  1998 
(M  =  213,  SD  =  19),  1999  (M  =  221,  SD  =  19)  and  2000  (M  =  223,  SD  =  18)  were 
compared  to  those  for  LBL  students  in  1 995  (M  =  202,  SD  =  1 7)  and  1 996  (M  =  199,  SD 
=  20).  National  means  for  USMLE  Step  1  scores  were  204  (1995),  205  (1996),  208 
(1997),  210  (1998),  213  (1999),  and  216  (2000).  Students  in  the  last  class  of  LBL  (1996) 
scored  significantly  lower  than  the  national  USMLE  Step  1  average  (p  <  .01).  After 
curriculum  conversion,  USMLE  Step  1  scores  for  PBL  students  were  equivalent  with  the 
national  mean  in  1997  and  1998,  and  significantly  greater  in  1999  (p  <  .001)  and  2000  (p 
<  .001).  According  to  the  methods  described  in  Chapter  3,  effect  size  estimates  were 
computed  for  each  PBL  class  versus  each  historical  control  group,  and  the  values  then 
were  averaged  to  produce  an  aggregate  effect  size  estimate  of  knowledge  achievement. 
A  standardized  effect  size  estimate  of  0.88  is  significant  because  95%C1  0.58  to  1.18 
does  not  contain  zero.  Despite  being  an  outlier  and  employing  a  historical  control, 
longitudinal  features  strengthen  the  results  of  this  study.  Curricular  conversion  was 
associated  with  achievement  gains,  immediately  rendering  students'  performance 
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comparable  and  outstripping  the  national  meati  within  three  years,  while  matriculating 
academic  and  demographic  characteristics  of  matriculates  remained  equivalent.  Because 
of  academic  baseline  equivalence,  change  in  the  results  of  USMLE  Step  1  scores  with  the 
implementation  of  problem-based  instruction  is  not  likely  due  to  alterations  in  class 
characteristics.  One  plausible  rival  explanation  for  these  results  could  be  the  Hawthorne 
effect.  Although  the  Hawthorne  effect  represents  a  threat  to  validity,  sustained  or 
improved  achievement  for  three  consecutive  years  makes  it  an  unlikely  explanation. 
These  results  suggest  cognitive  advantages  can  be  realized  relatively  rapidly  following 
conversion  to  problem-based  teaching  and  learning,  even  when  the  conversion  is  from  a 
traditional  curriculum. 
Examination  of  Select  Included  Studies 

Eisenstaedt,  Barry,  and  Glanz  (1990)  conducted  an  investigation  involving 
medical  students  from  three  consecutive  classes  to  examine  retention  of  knowledge 
acquired  during  the  hematology-transfusion  medicine  segment  of  a  second-year 
pathophysiology  course  via  problem-based  versus  the  traditional  lecture-based 
instruction.  Problem-based  instruction  was  facilitated  using  tutorial  groups  meeting  three 
times  a  week  for  about  two  hours.  Assessments  with  multiple-choice  questions  were 
conducted  at  the  end  of  the  course  and  again  with  a  similar  fomi  two  years  later  when 
students  were  seniors.  Lecture-based  students  initially  outperfonned  problem-based 
students  (M  =  80%,  SD  =  2%  vs.  M  =  67%,  SD  =  3%),  but  scores  virtually  were 
equivalent  when  assessment  was  repeated  two  years  later  (M  =  68%,  SD  =  2%  vs.  M  = 
65%,  SD  =  3%).  Investigators  concluded  initial  learning  may  be  poorer  in  students 
undergoing  problem-based  instruction,  but  they  appear  better  able  to  retain  knowledge 
(ES  =  0.46).  Effect  size  for  the  study  is  0.07.  Consistent  with  the  principles  advanced  by 
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some  social  construct! vists  (Cobb,  2002;  Savery  &  Duffy,  1995)  and  situational  learning 
theorists  (Brown  et  al,  1989,  Greeno,  1998;  Lave  &  Wenger,  1991),  these  data  suggest 
problem-based  learning  prompts  students  to  acquire,  process  and  represent  information  in 
a  manner  more  likely  to  be  retained  over  time.  Another  interpretation  of  this  differential 
forgetting  is  that  students  undergoing  lecture-based  learning  acquired  more  information 
to  forget  (Semb  &  Ellis,  1994).  These  results  are  not  idiosyncratic.  Miller  (1961) 
reported  poor  retention  of  biomedical  science  information  by  his  medical  students  who 
underwent  lecture-based  instruction  and  speculated  that  the  content  presented  via  teacher- 
centered  inethods  would  "decrease  from  memory  at  the  same  rate  as  nonsense  syllables" 
(p.  153).  Similarly,  Levine  and  Forman  (1973)  assessed  students  starting  a  neurology 
clerkship  (senior  year)  using  fifty  clinically  relevant  questions  derived  from  a  final  exam 
each  student  passed  in  their  first  year  of  lecture-based  medical  education.  Only  48%  of 
senior  medical  students  obtained  a  passing  grade.  Despite  potential  validity  threats  (e.g., 
history  and  maturation),  an  important  implication  of  these  data  is  that  the  knowledge 
achievement  advantage  realized  by  lecture-based  versus  problem-based  instruction  is 
transient.  Results  of  this  data  analysis  also  reinforce  the  superiority  of  longitudinal 
studies  to  assess  appropriately  the  impact  of  educational  interventions  rather  than  suffer 
susceptibility  to  inisleading  interpretations  associated  with  cross-sectional  research. 

Goodman  and  colleagues  (1991)  compared  curricular  effects  on  NBME  Part  I 
examination  scores  in  597  students  at  Rush  Medical  College  matriculating  in  classes  from 
1984  to  1988  (LBL  n  =  501;  PBL  n  =  72).   Discipline-bounded  lectures  and  laboratory 
sessions  primarily  drive  learning  in  a  lecture-based  curriculum.  Assessments  typically 
involve  multiple-choice  questions  designed  to  examine  student  "retention  of  discipline- 
specific  concepts  and  factual  content"  (p.  88).  Problem-based  learning  is  operationalized 
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during  the  first  two- years  of  medical  school  with  tutors  trained  to  use  Socratic  methods 
(term  employed  by  authors)  to  encourage  students  in  groups  of  six  meeting  twice  weekly 
to  generate  and  test  hypotheses  and  solve  problems.  Achievement  appraisal  of  PBL 
students  is  via  modified  essay  tests  and  "triple-jump  format"  assessments  aligned  with 
curricular  objectives  and  authentic  clinical  problems  as  discussed  in  Chapter  2.  This 
examination  fonnat  is  designed  to  "help  students  to  be  more  reflective  during 
evaluations"  (p.  88).  During  the  course  of  the  study,  six  students  transferred  from  PBL  to 
a  LBL  curriculum  and  one  student  transferred  from  PBL  to  the  LBL  curriculum.  Data 
from  these  students  were  censored.  Mean  NBME  Part  I  scores  were  controlled  for 
undergraduate  science  grade  GPA,  MCAT  scores,  age,  gender,  and  other  degrees  earned. 
The  authors  presented  adjusted  and  unadjusted  mean  data.  Unadjusted  effect  size 
computation  was  included  in  the  aggregated  data  analysis  for  this  research  according  to 
the  methodological  reasons  described  in  Chapter  3.  However,  there  were  no  significant 
differences  between  the  corrected  (M  =  489  vs.  M  =  480)  and  uncorrected  mean  data  (M 
=  489  vs.  M  =  485)  for  LBL  versus  PBL,  respectively.  The  first  attempt  failure  rate  for 
the  NBME  Part  1  examination  was  10.8%  (54/501 )  and  5.6%  (4/72)  for  LBL  and  PBL, 
respectively,  but  the  difference  was  not  significant.  Although  students  were  not 
randomly  assigned,  the  authors  employed  covariate  analyses  to  match  groups  statistically 
and  observed  no  significant  curricular  affect  on  the  acquisition  of  biomedical  science 
knowledge  through  the  initial  five-years  of  implementing  PBL.   In  2000,  Rush  Medical 
College  discontinued  the  problem-based  pathway  and  unified  the  curriculum  intending  to 
incorporate  the  "best  features"  of  both  cuiricula  (Zajtchuk,  McLaughlin,  Jacob,  & 
Rothenberg,  2000). 
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Mennin,  Friedman,  Skipper,  Kalishman,  and  Snyder  (1993)  reported  a  similar  trial 
comparing  NBME  Part  I  examination  scores  in  508  and  167  medical  students  undergoing 
LBL  and  PBL  tracks  at  the  University  of  New  Mexico  from  1983  to  1992.  Problem- 
based  learning  was  operationalized  using  the  reiterative  approach  advocated  by  Barrow 
(1986).  Tutors  were  trained  in  PBL  techniques  and  were  content-experts.  Medical 
students  in  the  PBL  curriculum  included  those  directly  accepted  to  the  track  or  those  who 
were  randomly  assigned  from  students  requesting  PBL.  NBME  Part  I  examination  scores 
were  significantly  higher  in  students  who  underwent  lecture-based  versus  probletn-based 
instruction  (M  =  504  vs.  M  =  456,  e  =  .0001,  ES  =  -0.61,  95%CI  -0.79  to  -0.43). 

As  discussed  earlier,  in  1992  NBME  Part  I  was  replaced  with  USMLE  Step  1. 
Presumably,  the  USMLE  assesses  similar  biomedical  science  learning  with  a  similar 
number  of  multiple-choice  questions,  but  incorporates  a  greater  percentage  of  questions 
designed  to  tap  higher-order  learning  (e.g.,  application,  synthesis,  and  evaluation)  rather 
than  biomedical  science  knowledge  exclusively  (O'Donnell  et  al.,  1993).  Theoretically, 
PBL  students  are  advantaged  by  the  shift  in  emphasis  of  licensing  examinations  from 
assessing  biomedical  science  knowledge  to  applied  knowledge.  The  opportunity  to  test 
this  postulation  is  presented  by  two  studies  at  the  same  institution  (Wake  Forest 
University  School  of  Medicine)  comparing  the  cunicular  effects  of  LBL  and  PBL  on 
biomedical  science  knowledge  assessed  with  NBME  Part  1  scores  (Richards  &  Cariago, 
1993)  and  USMLE  Step  1  scores  (Enarson  &  Can-igo-Lo,  2001).  Richards  and  Cariago 
observed  no  difference  in  NBME  Part  1  scores  between  LBL  (n  =  79,  M  =  209,  SD  =  18) 
and  PBL  (n  =  22,  M  =  212,  SD  =  21)  students  completing  the  examination  in  1991  (t  = 
.76,  E  =  .45,  ES  =  0.17).  Similariy,  Enarson  and  Carrigo-Lo  discovered  no  difference  in 
USMLE  Step  I  scores  between  LBL  (n  =  168,  M  =  214,  SD  =  20)  and  PBL  (n  =  588,  M 
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=  208,  SD  =  19)  students  completing  the  examination  eacli  year  from  1992  to  1998  (t  = 
1.32,  E  =  .21,  ES  =  -0.31).  Twenty  LBL  and  nineteen  PBL  students  failed  their  initial 
attempt  to  pass  the  USMLE  Step  1 .  Despite  an  increased  emphasis  on  the  apphcation  of 
biomedical  science  knowledge  to  clinical  problems,  the  sensitivity  of  USMLE  Step  1  to 
detect  relative  instructional  effects  has  been  questioned  and  is  discussed  later. 

In  one  of  a  few  randomized  studies,  Antepohl  and  Herzig  (1999)  revealed 
statistically  equivalent  overall  factual  knowledge  achievement  (ES  =  -0. 14,  95%CI  -0.49 
to  0.21)  for  German  medical  students  (University  of  Cologne)  undergoing  problem- 
based  (n  =  55)  versus  lecture-based  (n  =  57)  teaching  and  learning  in  a  pharmacology 
course,  but  PBL  student  performance  was  significantly  greater  on  a  short-essay 
component  (ES  =  0.37,  95%CI  0.02  to  0.72)  of  the  final  examination.  Problem-based 
instruction  was  operationalized  as  reiterative  according  to  Barrows'  taxonomy  (1986). 
Tutors  were  content  experts,  but  functioned  primarily  as  discussion  facilitators,  because 
primary  responsibility  for  generating  learning  issues  resided  with  the  students. 
Differential  effects  by  assessment  type  have  important  educational  implications. 
Compared  to  courses  taught  in  a  didactic  fomiat,  PBL  may  confer  a  unique  advantage  on 
assessments  requiring  elaborated  versus  dispersed  knowledge  (Bordage  &  Zacks,  1984). 
Results  of  this  data  analysis  support  the  theoretical  argument  that  higher  order 
assessments  are  more  appropriate  for  assessing  students  undergoing  problem-based 
teaching  and  learning  (Barrows,  2000).  Moreover,  the  effect  size  of  this  randomized  trial 
exceeds  the  mean  effect  size  suggesting  studies  using  weaker  designs  potentially  provide 
valid  and  useful  results. 

In  a  similarly  designed  randomized  trial  involving  third-year  German  medical 
students  (University  of  Essen),  Michel  and  colleagues  (2002)  compared  knowledge 
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achievement  in  students  wlio  underwent  a  pharmacology  course  with  problem-based  (n  = 
28)  versus  lecture-based  instruction  (n  =  56).  Problem-based  teaching  and  learning  was 
operationalized  in  a  manner  nearly  identical  to  the  previous  Gernian  study,  except  self- 
directed  learning  was  supplemented  with  optional  weekly  plenary  sessions  for  all 
students,  independent  of  instructional  fomiat.  The  authors  presented  no  data  regarding 
plenary  session  attendance.  Final  examination  scores  were  employed  to  compare  inter- 
group  factual  knowledge  achievement.  Out  of  thirty  multiple  choice  questions,  students 
who  underwent  problem-based  versus  lecture-based  instruction  made  fewer  mistakes  (M 
=  7.6,  SD  =  4.0  vs.  M  =  9.7,  SD  =  4.7  errors,  p  <  .05).  Computed  effect  size  of  0.48  is 
significant  because  95%C1  0.02  to  0.94  does  not  include  zero. 

In  the  same  study,  investigators  reported  another  data  set  involving  subsequent 
registrants  for  the  pharmacology  course.  These  students  were  permitted  to  select  either  a 
problem-based  (n  =  40)  or  a  lecture-based  (n  =  102)  instructional  format.  Interestingly, 
other  than  sampling  method,  course  content  and  instructional  methods  were  similarly 
operationalized,  but  an  effect  size  of  0.21  was  not  significant  because  95%CI  -0.16  to 
0.58  includes  zero.  Investigators  pooled  data  for  discussion.  When  data  are  pooled  using 
study  as  a  unit  of  analysis,  an  effect  size  of  0.36  is  significant  (95%CI  0.07  to  0.65). 
External  validation  was  accomplished  by  comparing  pooled  results  for  each  instructional 
format  with  scores  from  a  national  examination  given  after  the  third  year  of  medical 
school.  Students  who  underwent  problem-based  versus  lecture-based  scored  M  =  523 
(SD  =  76)  vs.  M  =  500  (SD  =  91 ,  respectively  in  phannacology;  and  M  =  493  (SD  =  79) 
vs.  M  =  480  (SD  =  95),  respectively  in  overall  examination.  Computed  effect  size  for 
pharmacology  achievement  is  0.28  (95%CI  -0.03  to  0.59)  and  overall  achievement  is  0.15 
(95%CI  -0.16  to  0.46).  The  investigators  do  not  mention  outcomes  vis-a-vis  differential 
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sampling  methods,  nor  do  they  present  data  to  help  account  for  the  varied  effects. 
However,  because  samphng  techniques  were  distinct,  each  trial  was  considered 
independent  for  current  analysis.  Consequently,  study  number  87  appears  twice  in  figure 
4-3. 

Data  comparing  self-directed  learning  (pooled  ES  =  1.63,  95%CI  1.17  to  2.09) 
and  attitude  toward  learning  (pooled  ES  =  0.72,  95%CI  0.30  to  1.14)  also  are  compared 
m  this  report.  These  outcome  data  also  are  treated  as  individual  studies  in  respective 
sections  of  this  chapter.  Successful  completion  of  medical  pharmacology  requires 
students  to  understand  basic  mechanisms,  but  also  to  memorize  voluminous  detailed  facts 
regarding  pharmaceutical  classifications  and  individual  agents.  Despite  an  obligatory 
memorization  burden,  problem-based  teaching  and  learning  fostered  superior 
pharmacology  knowledge  achievement. 

Kaufman  and  JVIann  (1998)  assessed  biomedical  science  achievement  and  found  a 
negative  effect  of -0.63  when  comparing  consecutive  cohorts  of  second  year  students 
who  underwent  an  inaugural  two  classes  (1996,  1997)  of  a  problem-based  curriculum  (n 
=  132)  with  those  in  the  last  class  (1995)  of  a  lecture-based  curriculum  (n  =  50)  at 
Dalhousie  University.  A  negative  effect  was  not  significant  because  the  95%CI  -0.96  to 
0  .30  included  zero.  There  were  no  differences  in  knowledge  test  scores  attained  between 
PBL  cohorts.  Knowledge  test  forniat  was  short-essay,  which  theoretically  should  favor 
PBL  students.  Cronbach's  alpha  for  tests  averaged  .74  and  inter-rater  reliability  was  .90. 
The  PBL  curriculum  aimed  to  employ  clinical  problems  "as  a  sfimulus  to  learning  the 
required  basic,  clinical  and  behavioral  science  content"  (p.  249).  Seven  to  eight  students 
grouped  three  times  weekly  for  two-hour  faculty-tutored  discussions  using  a 
"standardized  clinical  reasoning  process"  (p.  249)  characterized  by  the  authors  as  a 
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liypothetico-deductive  method.  Tutorials  were  supplemented  with  weelcly  lectures, 
laboratory  sessions,  and  other  formally  structured  teaching  and  learning  situations. 
Contrary  to  the  experience  of  Blake  and  colleagues  (2000),  curricular  conversion  at 
Dalhousie  University  proved  less  effective  with  respect  to  biomedical  science 
achievement  gain,  although  knowledge  achievement  was  assessed  with  distinctly 
different  instruments.  However,  Dalhousie  University  employed  an  assessment  that 
should  have  advantaged  students  undergoing  PBL  versus  LBL.  Unfortunately,  no 
substantive  differences  are  apparent  between  the  two  implementations  of  problem-based 
teaching  and  learning.  Clinical  achievement  measures  reported  (presented  later)  by  this 
group  reveal  statistical  parity  in  physicians-in-training  who  completed  LBL  and  PBL 
curricula  (Kaufman  &  Mann,  1998),  a  finding  that  suggests  the  functional  impact  of 
differential  learning  of  biomedical  sciences  vanishes  after  graduation  from  medical 
school.  These  results  support  the  conclusions  of  Eisentaedt  and  colleagues  (1990),  and  if 
further  substantiated  in  varied  contexts,  have  important  educational  implications.  For 
example,  some  biomedical  knowledge  believed  to  be  important  for  successful  clinical 
practice  may  not  be.  If  true,  biomedical  science  and  clinical  science  training  are 
misaligned.  Integrating  only  the  requisite  science  basic  to  authentic  clinical  problems 
may  better  facilitate  understanding  and  meinory  of  relevant  infomiation  (Cobb,  2002). 
This  pedagogical  approach  is  described  by  Cobb  as  "pragmatic  constructivism."  As 
discussed  in  Chapter  2,  pragmatic  constrtictivism  is  an  instructional  stance  that 
concurrently  addresses  the  process  of  student  learning  as  well  as  the  relevance  of  what  is 
learned. 

Dyke  and  associates  (2001)  compared  the  differential  effects  of  problem-based 
and  lecture-based  learning  in  second-year  medical  students  completing  a  thirteen  week 
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epidemiology  course.  Consenting  students  (n  =  80)  were  randomly  paired,  matched  for 
gender  and  characteristics  for  acceptance  to  medical  school,  then  randomly  assigned  to 
undergo  epidemiology  via  problem-based  or  lecture-based  instruction.  Non-consenting 
students  (n  =  56)  completed  lecture-based  epidemiology.  Problem-based  learning 
incorporated  small-group  tutorials  twice  a  week  for  at  least  two  hours  with  an  expert  tutor 
employing  the  reiterative  process  described  by  Barrows  (1986,  2000).  Tutors  were 
trained  in  the  techniques  of  PBL  and  met  regularly  during  the  term  to  discuss  the 
program.  Achievement  assessments  were  identical  for  each  instructional  group  and 
included  four  in-class  MCQ  examinations  and  a  take-home  test.  The  second  MCQ 
examination  and  take-home  assessment  were  designed  to  examine  depth  of  understanding 
and  critical  thinking.  There  were  no  significant  differential  effects  of  instructional 
formats  in  overall  knowledge  achievement  (ES  =  0.04,  95%CI  -0.33  to  0.41). 
Importantly,  inter-group  differences  did  not  manifest  in  assessments  of  depth  of 
understanding  and  critical  thinking.  Although  rating  take-home  assessments,  faculty 
claimed  that  compared  to  lecture-based  students,  PBL  students  provided  answers  that  are 
more  elaborate.  Lack  of  instructional  effect  on  achievement  may  have  been  confounded 
by  common  access  to  lecture  summaries  available  in  the  library.  Unfortunately,  potential 
"history"  threats  to  internal  validity  weaken  the  cognitive  interpretation  of  these  results. 
The  investigators  conclude,  however,  that  problem-based  and  lecture-based  instruction 
facilitated  equivalent  knowledge  gains,  but  PBL  students  reported  a  "far  richer  learning 
experience  than  did  their  counterparts  in  the  traditional  stream"  as  "medical  students 
using  PBL  think  their  understanding  of  underlying  principles  is  greater,  enjoy  the  course 
more,  and  emerge  from  it  more  enthusiastic  about  epidemiology  and  its  perceived 
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relevance  to  their  future  careers"  (p.  373)  These  latter  observations  emerged  from 
measures  of  attitude  toward  learning  reported  later  in  this  chapter. 
Examination  of  Select  Excluded  Studies 

Farquhar,  Haf,  and  Kotabe  (1 986)  reported  differential  NBME  Part  I  scores 
without  a  measure  of  variance  or  exact  p  value,  thus  standardized  effect  size  could  not  be 
computed.  The  PBL  (n  =  40)  and  LBL  (n  =  40)  groups  were  matched  academically 
(MCAT  scores)  and  demographically  (age,  gender,  minority  students)  in  an  institution 
with  twelve  years  experience  operating  parallel  cun-icular  tracks.  There  was  no  attrition 
in  either  group.  Scheduled  biomedical  science  lecture  time  over  fifty  weeks  for  LBL 
students  averaged  808  hours,  while  for  PBL  students  it  averaged  fifty-two  hours.  PBL 
students'  only  scheduled  lecture  hours  were  for  anatomy.  Students  in  the  PBL  track  meet 
four  hours  per  week  in  groups  of  twelve  with  two  faculty  members.  Mean  NBME  Part  I 
scores  favored  LBL  students  by  5.9,  but  the  difference  was  not  significant.  Despite 
approximately  fifteen  times  more  scheduled  biomedical  science  lecture  hours  for  LBL 
students,  PBL  students  exhibited  equivalent  biomedical  science  knowledge  achievement 
as  assessed  by  the  NBME  Part  I  examination. 

The  relative  effects  of  problem-based  and  lecture-based  instruction  were  reported 
by  Moore  (1991)  with  data  that  is  not  compatible  for  compufing  standardized  effect  size. 
For  two  years,  Harvard  Medical  School  operated  parallel  tracks  of  PBL  and  LBL.  Then 
m  1 987,  first  year  cun-icula  were  combined,  and  medical  students  who  selected  (n  =  54) 
PBL  were  compared  with  those  who  did  not  (n  =  81 )  as  they  underwent  a  first  year 
cuniculum  incorporating  both  PBL  and  LBL  courses.  Tliree  courses  were  PBL,  one 
course  was  a  mixture,  and  one  course  was  entirely  lecture-based.  Problem-based  courses 
included  no  more  than  one  lecture  per  day  and  relied  primarily  on  tutorial  discussion  of 
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problems  as  the  foundation  for  students'  decisions  about  what  to  study.  The  mixed 
course  covered  all  material  in  lecture  fonnat,  but  this  was  supplemented  with  problem- 
based  tutorials.  The  lecture-based  course  was  teacher-centered  and  syllabus  driven. 
Assessments  for  each  course  incorporated  a  combination  of  multiple-choice  and  essay 
questions.  In  PBL  courses,  problem  solving  also  was  part  of  assessment.  Assessments 
were  scaled  to  100.  After  each  course,  students  completed  an  evaluation.  There  were  no 
differences  in  average  achievement  for  individual  courses,  thus  I  pooled  data  for  the  five 
courses  between  students  who  requested  PBL  (83.3%)  versus  those  who  did  not  (83.5%). 
Importantly,  83%  of  students  who  expressed  an  initial  desire  to  be  in  a  traditional 
cuiTiculum  track  changed  their  preference  after  completing  one  PBL  course.  Eighty-five 
to  97%  of  students  who  selected  the  PBL  track  sustained  their  preference  as  they 
completed  all  five  courses.  Despite  atdtudes  toward  learning,  students  successfully 
adapted  to  the  instructional  pattern  as  evidenced  by  achievement.  Importantly, 
achievement  of  students  in  the  lowest  academic  quartile  for  the  year  was  not  influenced 
by  instructional  fonnat.  There  is  speculation  that  the  self-directed  learning  requirement 
associated  with  problem-based  instruction  disadvantages  lower  achieving  students 
(Blumberg  et  al,  1990;  Zimmerman  &  Lebeau,  2000).  Results  of  this  data  analysis 
suggest,  however,  that  students,  independent  of  academic  history,  perform  comparably 
well  whether  undergoing  lecture-based  or  problem-based  instruction,  but  tend  to  prefer 
the  teaching  and  learning  style  of  problem-based  instruction.  Consequently,  these  results 
contradict  the  belief  that  less  intellectually  competent  medical  students  perfonn  better  in 
a  teacher-centered,  lecture-based  environment.  Moreover,  these  data  suggest  curriculum 
wide  implementation  of  problem-based  instruction  may  not  be  required  to  realize 
educational  aims,  which  is  consistent  with  the  overall  findings  of  the  present  research.  A 
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potential  disadvantage  of  course  situated  problem-based  instruction  embedded  in  a 
traditional  curriculum  might  be  that  traditional  courses  would  erode  the  time  available  for 
collaboration  and  self-directed  learning,  ergo,  diminishing  the  problem-based  teaching 
and  learning  experience.  This  speculation  remains  to  be  examined  by  future  research. 
Way,  Biagi,  Clausen,  and  Hudson  (1999)  compared  knowledge  achievement  in 
four  consecutive  classes  (years  1992  through  1995)  of  medical  students  undergoing  one 
of  three  simultaneous  preclinical  (biomedical  science)  instructional  tracks  at  Ohio  State 
University.  These  tracks  included  a  lecture-discussion  pathway  (LDP,  n  =  603), 
independent-study  pathway  (ISP,  n  =  76),  and  a  problem-based  learning  pathway  (PBLP, 
n  =  101).  Large-group  discipline-based  lectures  supplemented  with  small-group 
discussions  are  principle  modes  of  learning  in  LDP.  Students  matriculating  in  ISP  self- 
learn  using  structured  reading  resources,  computer-based  resources,  and  diagnostic 
practice  assessments.  Students  in  PBLP  acquire  biomedical  science  knowledge  via  a 
small-group  tutorial  process  approximating  the  reiterative  model  described  by  Barrows 
(1986).  There  is  no  formal  academic  interaction  and  each  pathway  has  a  unique  library, 
study  area,  and  curriculum  administration  office.  Analysis  of  covariance  was  used  to 
compare  Step  1  scores  as  a  function  of  curriculum  and  examination  year  while 
controlling  for  academic  ability.  Students  in  PBLP  had  significantly  higher  scores  than 
those  in  either  ISP  or  LDP  (F  =  4.17,  df  =  2/776,  e<  .05).  No  significant  interaction  was 
observed.  Since  neither  mean  and  variance  data  nor  pair-wise  contrast  statistics  were 
presented,  the  standardized  effect  size  could  not  be  estimated.  The  cognitive  value  of 
collaboration  and  clinical  problem-focused  learning  processes  associated  with  problem- 
based  instruction  appear  to  be  uniquely  substantiated  by  this  study.  Given  the  widespread 
clamor  for  medical  education  reform  and  a  unique  opportunity  to  investigate  three 


161 

simultaneous  curricular  formats,  the  paucity  of  research  emerging  from  this  institution  is 
remarlcable. 

Clinical  Science  Achievement 
Hypothesis  3.  Aggregated  effect  size  estimates  for  clinical  science  achievement 
outcomes  in  students  and  physicians  who  undergo  medical  education  with  problem-based 
versus  lecture-based  teaching  and  learning  will  be  significantly  positive  because  the 
lower  limit  of  95%  confidence  interval  will  exceed  zero.  Measures  of  clinical  science 
achievement  include  standardized  examinations  (e.g.,  national  licensing  examinations) 
and  locally  developed  achievement  tests  incorporating  primarily  multiple-choice 
questions  or  multi-method  assessments  (e.g.,  combination  MCQ,  short  answer,  essay 
questions  and/or  practical  assessments).  Effect  size  estimates  and  associated  95% 
confidence  intervals  computed  from  clinical  science  achievement  outcomes  presented  in 
twenty-nine  studies  involving  7,340  participants  are  summarized  graphically  in  Figure  4- 
5.  Scrutinizing  the  plot  of  95%  confidence  intervals  for  each  effect  size  reveals  that 
thirteen  confidence  intervals  are  above  zero,  none  is  below  zero,  and  sixteen  include  zero. 
The  confidence  interval  for  all  but  four  studies  includes  the  combined  effect,  thus 
indicating  a  reasonably  consistent  effect  from  one  study  to  the  next.  Publication  bias  for 
clinical  science  achievement  outcomes  appears  unlikely  because  there  was  no  significant 
correlation  between  effect  size  estimates  and  sample  size  (Speamian's  rho  =  -.1 3,  g  = 
.52).  Students  and  physicians  undergoing  PBL  (n  =  2,545)  versus  LBL  (n  =  4,895) 
exhibited  significantly  greater  clinical  science  achievement,  ES  =  0.32,  95%C[:  0.27  to 
0.38.  If  computed  with  the  outlier  (ES  =  1 .58,  n  =  641 ),  the  mean  outcome  effect  size 
would  have  increased  to  0.39  (95%CI  0.0.34  to  0.44).  This  impressive  effect  size  may  be 
due  to  a  unique  operationalization  of  reiterative  PBL  and  is  discussed  later.  Also,  the 
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impact  of  the  outlier  is  amplified  by  weighting  the  effect  of  sample  size.  This  finding 
suggests  that  problem-based  instruction  under  average  conditions  represented  by  non- 
outlier  studies  can  be  expected  to  improve  clinical  science  achievement  for  a  typical 
student  from  the  50""  to  63'"''  percentile  of  the  lecture-based  instruction  population. 
According  to  Rosenthal's  (1979)  method,  1,209  additional  studies  with  a  mean  effect  size 
of  zero  are  required  to  reduce  the  combined  significance  of  clinical  achievement 
outcomes  to  less  than  05. 
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Study  Number 
Figure  4-5.  Plot  of  standardized  effect  sizes  and  associated  95%  confidence  intervals  for 
clinical  science  achievement.  Note.  Data  are  derived  from  29  independent 
studies  involving  7,340  participants.  Citation  for  study  number  is  listed  in 
Appendix  A.  Dashed  line  represents  mean  weighted  effect  size. 

Hypothesis  4.  Variation  in  clinical  science  achievement  effect  size  estimates  will 

exceed  random  error  as  evidenced  by  a  Q-statistic  at  the  .05  level  of  significance 

warranting  sub-group  analyses  to  test  for  potential  moderation  by  a  priori  defined 

outcome  measures,  methodological  features,  and  substantive  features  of  problem-based 
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learning.  The  frequency  distribution  of  standardized  effect  sizes  for  clinical  science 
achievement  outcomes  using  study  as  the  unit  of  analysis  are  represented  in  Figure  4-6. 
The  results  of  a  homogeneity  analysis  suggest  the  distribution  of  effect  sizes  for  clinical 
achievement  are  greater  than  would  be  expected  entirely  from  sampling  en'or,  Qr(28)  = 
162.8,  2<  .00001).  Since  homogeneity  hypothesis  is  rejected,  the  implication  is  that 
further  exploration  is  warranted  to  discover  whether  study  features  are  operating  to 
moderate  variance  of  effect  sizes.  The  distribution  of  within-group  and  betvveen-group 
variances  of  effect  size  estimates  associated  with  outcome,  methodological,  and 
substantive  features  are  summarized  in  Tables  4-6,  4-7  and  4-8. 
9r 


Figure  4-6.  Frequency  distribution  of  standardized  effect  sizes  for  clinical  science 

achievement.  Expected  normal  distribution  curve  is  illustrated.  Note.  Data 
are  derived  from  29  studies  with  effect  sizes  involving  7,340  participants. 
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Outcome  features.  Clinical  science  achievement  effect  size  estimates  varied 
with  the  type  of  assessment  instrument  and  are  summarized  in  Table  4-6.  Twelve  of 
twenty-nine  reports  provided  multiple  outcome  measures  for  clinical  science 
achievement.  Outcome  source  was  a  significant  predictor,  Qb(1)  =  6.9,  p<  .001.  Studies 
contrasting  instructional  effects  with  non-licensing  assessments  reported  significantly 
greater  effects  (ES  =  0.37)  than  studies  using  licensing  tests  (ES  =  0.24).  However,  a 
significant  amount  of  variability  remained  within  individual  groupings:  licensing  tests, 
Qwi  =  113.3,  g<  .001,  and  non-licensing  assessments,  Qw,  =  91.3,  g  =  .004.  Therefore, 
licensing  and  non-licensing  assessments  were  stratified  according  to  type  and  analyzed. 
The  effect  size  for  clinical  achievement  was  significantly  positive  when  assessed  with  all 
measures  except  the  Medical  Council  of  Canada  Qualifying  Examination  Part  II,  ES  =  - 
0.04,  95%CI  -0.37  to  0.30,  which  was  reported  in  one  study.  Measurement  type  effect 
size  variance  was  dominated  by  USMLE  Step  2  (ES  =  0.39,  51%  of  Qw).  but  represented 
only  five  independent  samples.  However,  effect  size  for  USMLE  Step  2  was 
significantly  greater  than  that  observed  for  NBME  Part  II  (ES  =  0.12).  In  1992,  NBME 
Part  II  was  replaced  with  USMLE  Step  2.  The  USMLE  Step  2  is  designed  to  assess 
similar  clinical  science  knowledge  with  a  similar  number  of  multiple-choice  questions, 
but  incorporates  a  greater  percentage  of  questions  deigned  to  tap  higher-order  learning 
rather  than  factual  knowledge  exclusively  (O'Donnell  et  al.,  1993).  Theoretically.  PBL 
students  are  advantaged  by  a  shift  in  objectives  and  item  construction  of  licensing 
examinations.   By  implication,  the  present  findings  lend  support  to  the  construct  validity 
ofUSMLEStep2. 
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Table  4-6.  Mean  effect  sizes  and  variance  distribution  for  clinical  science  achievement  as 
a  function  of  outcome  measures 


N 

n 

ES 

95%CI 

Homogeneity 

Variables 
Outcome  source 

Lower 

Upper 

Qb' 

Qw," 

6.9** 

Licensing 

17 

4,981 

.24 

.17 

.31 

113.3** 

Non-licensing 

24 

4,040 

.37 

.30 

.44 

91.3** 

Outcome  type 

Licensing 

13.4** 

NBME  Part  11 

8 

1,977 

.12 

.01 

.22 

6.1 

USMLE  Step  2 

5 

1,906 

.39 

.28 

.49 

91  2** 

NBME  Part  III 

1 

416 

.33 

.11 

.55 

00 

MCCQE  Part  I 

2 

542 

,20 

.02 

.39 

0  1 

MCCQE  Part  II 

1 

140 

-.04 

-.37 

.30 

00 

Non-licensing 

11.5* 

NBME-II  Shelf-boards 

5 

1,384 

.45 

.34 

.56 

36.2** 

Multimethod  tests 

9 

772 

.48 

.30 

.65 

20  5* 

MCQ  tests 

4 

257 

.46 

.21 

.72 

6.8 

Supervisor's  rating 

6 

1,627 

.29 

.13 

.45 

16.4** 

=  number  of  weighted  effect  sizes  (ES)  averaged;  n  =  number  of  subjects  and 
sums  to  more  than  7,440  due  to  redundant  measures.  If  95%  confidence  interval  (CI) 
does  not  contain  zero,  ES  is  significant  at  .05  level.  Qh  =  between-group  variance;  Qwi  = 
withm-group  variance  for  each  feature  sub-group.  NBME  =  National  Board  of  Medical 
Examiners  Part  I;  USMLE  =  United  States  Medical  Licensing  Examination  Step  2,  which 
replaced  NBME  Part  II  in  1992;  NBME  Part  III  was  replaced  by  USMLE  Step  3  in  1992, 
but  no  comparative  research  permitted  calculation  of  effect  size;  MCCQE  =  Medical 
Council  of  Canada  Qualifying  Examination,  which  incorporates  Part  I  and  Part  II  in  the 
licensing  process;  NBME-II  =  National  Board  of  Medical  Examiners  Shelf-board 
examinations,  which  are  given  during  Family  Medicine,  Internal  Medicine,  Obstetrics- 
Gynecology,  Pediatrics,  Psychiatry  and  Surgery  clerkships.  Multimethod  tests  =  varied 
assessment  strategies  including  multiple  choice  questions  (MCQ),  short  answer,  essay, 
matching  items  and  practical  examinations  requiring  interaction  with  standardized 
pafients  or  simulations.  "Significance  indicates  rejection  of  hypothesis  of  between-group 
homogeneity.    Significance  indicates  rejection  of  hypothesis  of  within-group 
homogeneity. 
*E<.05,  **E<.01. 

Clinical  science  achievement  was  significantly  greater  when  problem-based 
instructional  efficacy  was  measured  with  non-licensing  assessments,  particulariy,  NBME- 
II  Shelf-boards  (ES  =  .45),  locally  constructed  multi-method  tests  [(ES  =  0.48)  and  MCQ 
(ES  =  0.45)],  and  supervisor  ratings  (ES  =  .29).  Therefore,  to  examine  methodological 
and  substantive  study  features  as  potential  moderators  of  instructional  effect,  further 
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analyses  of  non-licensing  measures  were  conducted  to  discover  grouping  patterns  with 
relatively  stable  variance  and  homogeneity.  Grouping  independent  samples  comparing 
scores  from  NBME-II  Shelf-boards,  multi-method,  and  MCQ  assessments  produced 
similar  effect  size  estimates  and  no  between-group  significance  (Qb  =  0.003,  g^=  .96). 
Although  there  is  significant  variance  (Qw  =  62.6,  g  <  .001),  grouping  reduces  by  65% 
the  cumulative  within-group  variance  for  outcome  measure  types  (Qw  =  178.9). 
Therefore,  assuming  sufficient  variance  is  explained  by  characteristics  of  this  relatively 
homogeneous  grouping,  methodological  and  substantive  features  were  analyzed  from 
instructional  contrasts  in  sixteen  studies  using  NBME-Il  Shelf-boards  (N  =  5)  or  locally 
constructed  multi-method  (N  =  9),  or  MCQ  (N  =  4)  assessments.  Using  the  study  as  the 
unit  of  analysis,  the  mean  effect  size  estimate  for  these  outcome  types  of  0.47  is 
significant  because  95%CI  .38  to  .56  does  not  include  zero.  There  is  no  evidence  of 
publication  bias  for  this  reduced  sample  (Speamian's  rho  =  .04,  p  =  88).  Moreover, 
according  to  Rosenthal's  (1979)  method,  454  additional  samples  with  a  mean  effect  size 
of  zero  are  required  to  reduce  the  significance  of  this  reduced  data  set  of  clinical  science 
achievement  outcomes  to  less  than  .05.  Because  external  validity  may  be  jeopardized  by 
reactive  or  interaction  effects  of  assessment,  caution  is  urged  regarding  the  interpretation 
of  results  derived  from  this  reduced  data  set,  as  outcome  measures  closely  align  with  the 
objectives  of  problem-based  rather  than  those  of  lecture-based  instruction. 

Methodological  features.  Variances  in  methodological  features  are  summarized 
in  Table  4-7.  Subjects  were  randomized  in  only  four  studies  (ES  =  0.29),  and  the  mean 
effect  size  estimate  was  statistically  greater  when  studies  compared  intact  groups  (ES  = 
0.51,  E  =  .05).  Baseline  equivalence  was  reported  in  all  but  one  study.  Control  group 
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characteristics  did  not  moderate  effect  size  estimates.  Qb(1)  =  3.1,  p  =  .08.  The 
differential  effects  of  instructional  approach  were  not  significantly  different  when 
compared  to  a  historical  control  group  (ES  =  0.55),  or  when  compared  to  a  parallel 
control  group  (ES  =  .39).  However,  the  distribution  of  studies  was  dominated  by  parallel 
group  designs  (81%  of  studies).  Whether  findings  were  from  published  journals  or  from 
dissertations  had  no  influence  on  effect  size  variance,  as  journal  publications  (ES  =  .47) 
accounted  for  99%  of  Qw.  Indeed,  the  only  significant  negative  effect  size  (ES  =  -.39) 
was  reported  in  a  dissertation  involving  research  at  Baylor  Medical  School  employing 
NBME-11  Shelf-boards  and  a  locally  constructed  multiple-choice  assessment.  A  unique 
aspect  of  this  program  was  1 .5  versus  2-years  of  required  pre-clinical  biomedical 
education  for  both  PBL  and  LBL  curricular  tracts.  Whether  shorter  duration  of 
biomedical  science  education  uniquely  disadvantages  students  undergoing  problem-based 
instruction  remains  to  be  examined  in  future  research.  A  decade  of  publication  did  not 
significantly  influence  effect  size  estimates  with  thirteen  of  sixteen  samples  reported  in 
the  1990s  (ES  =  0.49). 

Substantive  features.  Variance  analyses  of  substantive  features  are  summarized 
in  Table  4-8.  Greater  clinical  science  achievement  was  revealed  when  problem-based 
instniction  was  employed  for  an  entire  course  (ES  =  0.51)  versus  when  curriculum-wide 
(ES  =  0.49)  or  for  part  of  a  course  (ES  =  0.16),  but  the  differences  were  not  significant. 
However,  instructional  effect  significantly  favored  PBL  when  employed  curriculum-wide 
or  for  an  entire  course,  but  not  when  used  for  part  of  a  course.  The  taxonomy  of  PBL, 
Qb(2)=  7.9,  E  =  .02,  and  problem-type,  Qb(2)  =  9.0,  p  =  .01 ,  significantly  moderate  effect 
size  estimates  for  clinical  achievement.   Reiterative  PBL  (ES  =  0.51)  and  problem-based 
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Tabic  4-7.  Mean  effect  sizes  and  variance  distribution  for  clinical  science  achievement  as 
a  function  of  methodological  features 


95%C1 


Homogeneity 


Variables 

N 

ES 

Lower 

Upper 

Qb 

Qwi 

Sample  design 

4.0* 

Random  assignment 

4 

.29 

-.02 

.61 

4.4 

Intact  groups 

12 

.51 

.41 

.62 

45.0 

Baseline  Equivalence 

1.0 

Yes 

15 

.49 

.39 

.60 

51.9 

No 

1 

.45 

.12 

.78 

0.5 

Control  group 

3.1 

Parallel  curriculum  /  course 

13 

.39 

.25 

.53 

30.7 

Historical  curriculum  /  course 

3 

.55 

.28 

.81 

19.4 

Reliability 

3.0 

Yes 

4 

.37 

.14 

.60 

4.5 

No 

12 

.53 

.41 

.65 

45.8 

Validity 

1.4 

Yes 

4 

.41 

.17 

.63 

5.8 

No 

12 

.51 

.39 

.63 

46.1 

Publication  source 

0.0 

Journal 

15 

.47 

.38 

.56 

53.3 

Dissertation 

1 

-.39 

-.78 

-.01 

0.0 

Publication  year 

0.3 

1980-1989 

1 

.22 

-.39 

.82 

0.0 

1990-1999 

13 

.49 

.37 

.62 

51.2 

2000-2002 

2 

.44 

.30 

.59 

1.1 

Note.  N  =  number  of  weighted  effect  sizes  (ES)  averaged.  If  95%  confidence  interval 
(CI)  does  not  contain  zero,  ES  is  significant  at  .05  level.  Qij  =  between-group  variance; 
Qwi  =  within-group  variance  for  each  feature  sub-group.  "Significance  indicates  rejection 
of  hypothesis  of  between-group  homogeneity. 
*2<.05,  **E<.01. 

instruction  (ES  =  0.52)  showed  equally  significant  effecfiveness.  Only  two  studies 

employed  a  modified  case-method  version  of  problem-based  learning  (ES  =  0.09),  and 

the  effect  was  not  significant.  Problem-based  instruction  operationalized  with  combined 

ill-structured  and  structured  problems  (ES  =  0.65)  produced  the  largest  effect  size. 

Magnitude  of  effect  size  was  similar  when  problems  were  either  ill-structured  (ES  =  0.37) 

or  structured  (ES  =  0.40),  but  only  two  studies  employed  the  latter.  Whether  origin  of 

learning  issues  resided  solely  with  students  (ES  =  0.45)  or  students  and  faculty  jointly 

(ES  =  0.47)  had  no  affect  on  the  magnitude  of  reported  effect  size. 
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Tabic  4-8.  Mean  effect  sizes  and  variance  distribution  for  clinical  science  achievement  as 
a  function  of  substantive  features 


N 

ES 

95%CI 

Homo 

geneity 

Variables 

Lower 

Upper 

Qb' 

Qw, 

Scope  of  instruction 

5.2 

Curriculum 

4 

.49 

.12 

.82 

17.5 

Entire  course 

9 

.51 

.39 

.63 

27.7 

Part  of  a  course 

3 

.16 

-.46 

.78 

2.9 

PBL  taxonomy 

7.9* 

Reiterative  PBL 

7 

.51 

.38 

.64 

37.5 

Problem-based 

7 

.52 

.28 

.76 

7.4 

Modified  case-method 

2 

.09 

-.18 

.37 

0.5 

Problem  type 

9.0* 

Ill-structured 

7 

.37 

.22 

.52 

7.9 

Structured 

2 

.40 

.14 

.66 

2.8 

Mixed 

7 

.65 

.47 

.84 

33.6 

Learning  issues  responsibility 

0.03 

Student  generated 

7 

.45 

.31 

.60 

13.2 

Student-faculty  generated 

8 

.47 

.30 

.64 

39.3 

Tutor  expertise 

7.2* 

Content  expert 

10 

.54 

.42 

.65 

32.6 

Peer 

2 

.06 

-.29 

.40 

0.4 

Self-directed  learning 

18.2** 

Student  centered 

9 

.42 

.29 

.55 

21.5 

Plus  mandatory  lectures 

5 

.70 

.48 

.92 

13.4 

Plus  elective  lectures 

2 

.08 

-.18 

.34 

0.3 

Note,  N  =  number  of  weighted  effect  sizes  (ES)  averaged.  If  95%  confidence  interval 
(CI)  does  not  contain  zero,  ES  is  significant  at  .05  level.  Q,,  =  between-group  variance; 
Qw,  =  within-group  variance  for  each  feature  sub-group.  N  varies  among  independent ' 
variables  because  some  features  could  not  be  ascertained.  "Significance  indicates 
rejection  of  hypothesis  of  between-group  homogeneity. 
*E<.05,  **g<.OI. 

Tutor  expertise  was  a  significant  predictor  of  effect  size  estimates,  Qb(  I)  =  7.2,  e 
=  .03.  histructional  contrasts  with  content  expert  tutors  were  associated  with  significantly 
greater  effects  (ES  =  0.54)  than  those  with  peer  tutors  (ES  =  0.06).  However,  distribution 
of  studies  and  variance  (99%  of  Qw)  heavily  favored  tutors  with  content  expertise; 
unfortunately,  four  studies  did  not  provide  information  regarding  tutor  expertise. 
Characteristics  of  self-directed  learning  significantly  moderate  effect  size  estimates, 
Qb(2)  =  1 8.2,  E  <  .001 .  Self-directed  learning  orchestrated  principally  by  students  (ES  = 
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0.42)  and  when  supplemented  with  mandatory  lectures  (ES  =  0.70)  facihtated 

significantly  positive  clinical  science  achievement.  When  self-directed  learning  was 

augmented  with  elective-lectures,  an  effect  size  of  .08  was  not  significant. 

Table  4-9  presents  the  correlation  matrix  of  hypothesized  positive  predicative 

dimensions  for  PBL  efficacy  as  discussed  in  Chapter  2.  Tutor  expertise  was  the  only 

substantive  feature  correlated  with  effect  size  at  a  significance  level  of  .10  level  or  less. 

Interpretation  of  this  finding  is  severely  limited  because  ten  of  twelve  studies  reporting 

tutor  expertise  indicated  using  content  experts.  There  was  significant  multicolinearity 

between  substantive  features  including:  between  PBL  taxonomy  and  both  learning  issues 

responsibility  (r  =  .32,  p  =  .  1 0)  and  self  directed  learning  (r  =  .33,  p  =  .08);  and  learning 

issues  responsibility  and  self-directed  learning  (r  =  .67,  p  <  .0001). 

Table  4-9.  htercorrelations  in  PBL  dimensions  for  clinical  science  achievement 

Variable  A  R  r  n  F 


-.07 

.06 

.42* 

.09 

-.14 

.32* 

.21 

.33* 

.20 

-.12 

.06 

.30 

.67* 
.12 

Mean  effect  size  estimate  -.07 

A.  PBL  taxonomy 

B.  Problem  type 

C.  Learning  issues  responsibility 

D.  Tutor  expertise 

E.  Self-directed  learning 

Note.  Date  are  derived  from  16  studies,  except  for  tutor  expertise,  which  was  reported  in 
13  studies.  PBL  taxonomy  =  reiterative  or  not;  Problem  type  =  ill-structured  or  not; 
Learning  issues  responsibility  =  student  derived  or  not;  Tutor  expertise  =  content  expert 
or  not;  Self-directed  learning  =  student  directed  or  not. 

*p<.10. 

Examination  of  outlier  study 

Remmen  and  colleagues  (2001)  studied  clinical  science  achievement  in  medical 
students  during  their  last  four  years  of  training  at  Maastricht  University  (a  PBL  school) 
and  three  traditional  Universities  at  Groningen,  Antwerp,  and  Ghent.  Medical  school  in 
Holland  (Maastricht  and  Groningen)  and  Belgium  (Antwerp  and  Ghent)  requires  six  and 
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seven  years,  respectively.  Students  were  invited  to  complete  a  132-item  written  clinical 
achievement  assessment.  Face  and  content  validity  of  the  instrument  were  examined  by 
twenty  senior  faculty.  Items  identified  as  problematic  by  three  or  more  reviewers  were 
discarded.  Construct  validity  previously  was  tested  by  correlating  a  clinical  achievement 
assessment  score  and  a  score  for  an  OSCE  (objective  structured  clinical  examination) 
designed  to  assess  hands-on  performance  in  basic  clinical  skills.  The  OSCE  is  a  12- 
station  assessment  with  simulated  clinical  problems.  Correlation  for  the  two  scores  was 
0.87  (Ghent)  and  0.64  (Antwerp).  Maastricht  University  employs  reiterative  PBL  that  is 
operationalized  with  content  expert  tutors  and  collaborating  students  generating  learning 
issues  to  direct  individual  and  shared  clinically  relevant  biomedical  science  learning. 
Particular  content  achieveinent  is  assessed  using  a  triple-jump  method  and  general 
achievement  is  assessed  with  annual  progress  testing  as  discussed  in  Chapter  2.  Although 
students  were  not  randomly  assigned  to  each  curriculum,  students  accepted  for  medical 
school  in  Holland  (Maastricht  and  Groningen)  are  assigned  to  particular  schools  by  a 
national  lottery.  Clinical  achievement  data  were  pooled  and  standardized  effect  size  was 
computed  for  summary.  Standardized  effect  size  estimates  for  each  cohort  (academic 
level)  ranged  from  0.80  to  2.30.  A  standardized  effect  size  of  1.59  for  the  study  was 
significant  because  a  95%  confidence  interval  from  1.38  to  1.80  does  not  include  zero. 
Because  in  this  instance  problem-based  instruction  operationalizes  teaching  and  learning 
processes  and  assessment  as  a  continuum,  measurement  bias  may  have  favored  PBL 
studems.  Recognizing  external  validly  is  constrained  by  the  possible  reactive  effects  of 
experimental  arrangements  and  assessment;  methodological  robustness  adds  substantial 
inferential  power  to  this  research.  Whether  the  unique  implementation  of  problem-based 
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teaching  and  learning  at  Maastricht  University  would  be  equally  efficacious  elsewhere 
warrants  further  research.  Clearly,  the  implication  of  this  research  is  when  educational 
objectives  and  assessments  are  congruent,  the  relative  effects  of  instruction  emerge. 
Examination  of  select  included  studies 

Richards  and  associates  (1996)  conducted  a  longitudinal  study  comparing 
clinical  achievement  in  medical  students  undergoing  a  common  third-year  internal 
medicine  clerkship  at  Wake  Forest  University  Medical  School  from  1991  through  1995. 
The  students'  pre-clinical  education  was  either  LBL  (n  =  364)  or  PBL  (n  =  88).  Clinical 
achievement  was  assessed  with  internal  medicine  subject  NBME-II  Shelf-board 
examination  scores  and  supervisor  ratings  of  clinical  performance.  Intergroup  contrasts 
were  controlled  with  ANCOVA.  NBME-11  Shelf-board  scores  for  students  who 
underwent  LBL  (M  =  489,  SD  =  97)  and  PBL  (M  =  497,  SD  =  97)  were  not  different  (g  = 
.80);  effect  size  was  .07  with  a  95%C1  from  -0.15  to  0.30.  Dimensions  (and  effect  sizes) 
of  a  clinical  rating  scale  included:  (a)  amount  of  biomedical  science  knowledge  (.54);  (b) 
take  history  and  physical  (.46);  (c)  derive  differential  diagnosis  (.49);  and  (d)  organize 
and  express  information  (.42).  A  pooled  effect  size  estimate  for  clinical  rating 
dimensions  is  0.48,  which  is  significant  because  the  95%CI  from  0.25  to  0.71  does  not 
include  zero. 

Similar  results  were  observed  by  Distlehorst  and  Robbs  (1998)  who  compared 
clinical  achievement  for  three  consecutive  years  in  students  who  underwent  preclinical 
medical  education  in  a  reiterative  PBL  curriculum  (n  =  47)  with  those  in  a  parallel-track 
lecture-based  curriculum  (n  =  154)  at  Southern  Illinois  University  School  of  Medicine. 
Clinical  achievement  was  assessed  with  USMLE  Step  2  scores,  clinical  practice 
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examination  (CPX)  scores,  and  overall  performance  ratings  for  third-year  clerlcships 
including  Internal  Medicine,  Surgery,  Obstetrics/Gynecology,  Psychiatry,  Pediatrics,  and 
Family  and  Community  Medicine.  The  CPX  included  1 5-stations  with  individual  patient 
cases  (trained  surrogates).  Scores  for  CPX  derived  from  exhibiting  competencies  in 
differential  diagnosis,  data  interpretation  and  management,  and  final  diagnosis. 
Clerkship  rating  dimensions  included  achievement  in  history  and  physical  examination, 
diagnosis  and  patient  management,  technical  and  procedural  skills,  background 
knowledge,  knowledge  of  discipline  based  on  performance,  knowledge  of  discipline 
based  on  tests,  application  of  knowledge,  self-directed  learning,  interpersonal 
relationships,  and  motivation/dependability/responsibility.  Analysis  of  covariance  was 
employed  to  control  differential  demographic  and  academic  characteristics.  Standardized 
effect  sizes  were  computed  and  the  significance  quantified  with  p  values.  Mean  USMLE 
Step  2  scores  for  students  who  underwent  PBL  (M  =  208)  versus  LBL  (M  =  199)  were 
different  (ES  =  0.39,  p  =  .02).  There  were  no  intergroup  differences  in  CPX  scores  (ES  = 
0.30,  p  =  .07).  However,  overall  clerkship  rating  was  greater  in  PBL  versus  LBL 
students  (ES  =  0.50,  p  =  .003).  Study  as  the  unit  of  analysis  effect  size  estimate  is  0.45 
and  is  significant  because  the  95%CI  0.12  to  0.78  does  not  include  zero.  The 
implications  of  these  results  are  significant  because  in  this  setting,  problem-based 
instruction  is  operationalized  in  a  manner  closely  aligned  with  the  principles  of  education 
discussed  in  Chapter  2.  Moreover,  other  than  USMLE  Step  2,  assessment  of  student 
achievement  is  congruent  with  the  aims  of  medical  education  generally,  and  problem- 
based  instruction  particularly,  including  a  self-assessment,  peer-assessment,  tutor- 
assessment,  and  an  end-of-unit,  performance-based  assessment.  This  latter  assessment 
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includes  a  one-on-one  oral  examination  of  key  concepts  for  content  unit.  Instructional 
sensitivity  of  CPX  may  be  blunted  by  complexities  encountered  in  the  context  of  patient- 
problems  versus  pencil-and-paper  problems.  Whether  pcncil-and-paper  problems 
provide  authentic  problem-context  requires  further  research. 

Doucet,  Purdy,  Kaufman,  and  Langille  (1998)  compared  clinical  achievement 
gain  in  practicing  physicians  in  Halifax,  Nova  Scotia  undergoing  a  continuing  medical 
education  (CME)  course  focusing  on  headache  diagnosis  and  management.  Similar  core 
content  was  presented  via  problem-based  or  lecture-based  format.  Problem-based 
learning  look  place  in  small  groups  with  discussions  guided  by  authentic  patient  cases 
and  facilitated  by  expert  neurologists  with  extensive  PBL  experience.  Groups  retained 
the  same  tutor  throughout  the  course  and  convened  for  times  equivalent  to  lecture 
sessions.  The  lecture-based  format  included  three  weekly  90-minute  lectures  and  30- 
minute  discussions.  The  same  neurologists  lectured  and  tutored.  Pre-  and  post-test 
scores  were  obtained  immediately  before  and  after  the  three-month  course.  Assessment 
included  40-multiple  choice  questions  designed  by  a  neurologist  expert  in  headaches. 
Validation  of  content  was  provided  by  an  independent  neurologist.  Two  family 
practitioners  piloted  the  test  to  assess  practice  relevance.  A  second  assessment  was  given 
to  participants  three  months  after  course  completion.  Study  participants  were  drawn  from 
33%  responses  to  a  CME  needs  assessment  survey  (127/358).  Problem-based  instruction 
participants  (n  =  38)  included  74%  females  and  the  group  averaged  twelve  years  of 
practice.  Lecture-based  instruction  participants  (n  =  49)  included  49%  females  and 
averaged  seventeen  years  of  practice.  Data  were  evaluated  with  an  analysis  of  covariance 
using  the  pre-test  score  as  covariate.  Despite  the  differential  distribution  of  gender  and 
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practice  experience,  there  were  no  intergroup  differences  in  knowledge  pre-test  scores  (p 
=  .69).  However,  PBL  participants  had  significantly  higher  post-test  scores  (p  =  .05)  and 
three-month  follow-up  test  scores  (p  =  .001).  A  limitation  to  follow-up  data  was  an 
attrition  rate  of  43%  and  47%  for  PBL  and  LBL  participants,  respectively. 
Notwithstanding  a  similar  attrition  rate,  the  mean  assessment  score  for  the  PBL  group 
was  25%  greater  and  suggests  retention  of  knowledge,  even  knowledge  considered 
valuable  for  practice,  is  better  retained  when  learned  in  a  relevant  context. 
Problem  Solving  Achievement 
Hypothesis  5.  Aggregated  effect  sizes  for  problem-solving  skill  outcomes  in 
students  and  physicians  who  undergo  medical  education  with  problem-based  versus 
lecture-based  teaching  and  learning  will  be  significantly  positive  because  the  lower  limit 
of  95%  confidence  interval  will  exceed  zero.  Problem  solving  skills  were  defined  as 
data-gathering  skills,  accuracy  of  explanation,  coherence  of  explanation,  use  of  science 
concepts  linking  causes  and  working  (tentative)  hypotheses,  diagnostic  accuracy,  and/or 
treatment  accuracy.  Effect  size  estimates  and  associated  95%  confidence  intervals 
computed  from  problem-solving  skills  outcomes  presented  in  ten  studies  involving  1,236 
participants  are  summarized  graphically  in  Figure  4-7.  Examining  the  plot  of  95% 
confidence  intervals  for  each  effect  size  reveals  four  confidence  intervals  above  zero, 
none  below  zero,  and  six  include  zero.  Students  and  physicians  undergoing  PBL  (n  = 
592)  versus  LBL  (n  =  644)  exhibited  significantly  better  problem-solving  skills,  ES  = 
0.19,  95%C]  0.03  to  0.34.  This  finding  suggests  problem-based  instruction  under 
average  condifions  would  be  expected  to  improve  the  problem  solving  skills  of  a 
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Figure  4-7.  Plot  of  standardized  effect  sizes  and  associated  95%  confidence  intervals  for 
problem  solving  achievement.  Note.  Data  are  derived  from  ten  independent 
studies  involving  1 ,236  participants.  Citation  for  study  number  is  listed  in 
Appendix  A.  Dashed  line  represents  mean  weighted  effect  size. 

typical  learner  in  a  lecture-based  solving  instruction  population  from  the  50"'  to  58"' 
percentile.  Publication  bias  for  problem  solving  outcomes  is  unlikely  because  there  was 
no  significant  correlation  between  effect  size  estimates  and  sample  size  (Spearman's  rho 
=  -.07,  E  =  .81).  According  to  Rosenthal's  (1979)  method,  eighty-one  additional  studies 
with  a  mean  effect  size  of  zero  are  required  to  reduce  the  combined  significance  of 
problem  solving  outcomes  to  less  than  .05. 

Hypothesis  6.  Variation  in  problem  solving  achievement  effect  sizes  will  exceed 
random  error  as  evidenced  by  a  Q-statistic  at  the  .05  level  of  significance  warranting  sub- 
group analyses  to  test  for  potential  moderation  by  a  priori  defined  outcome  measures, 
methodological  features,  and  substantive  features  of  problem-based  teaching  and 
learning.  The  frequency  distribution  of  standardized  effect  size  estimates  for  problem- 
solving  outcomes  included  in  the  analyses  is  represented  in  Figure  4-8.  Results  of 
homogeneity  analysis  suggest  distribution  of  effect  size  estimates  for  problem  solving  are 
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not  greater  than  would  be  expected  entirely  from  sampling  error,  Qt(9)  =  8. 1 ,  g  =  .53. 
Therefore,  the  homogeneity  hypothesis  is  not  rejected  and  the  sampling  error  alone 
explains  this  magnitude  of  variance  in  problem  solving  skills  effect  sizes.  Since  variance 
is  statistically  homogeneous,  the  mean  of  effect  size  estimates  is  considered  to  be  an 
accurate  reflection  of  the  relative  superiority  of  problem-based  versed  lecture-based 
instruction  for  promoting  problem-solving  success  independent  of  outcome  measures, 
research  methodology,  and  the  substantive  dimensions  of  problem-based  teaching  and 
learning  study  features  (Hedges,  1994). 
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Figure  4-8.  Frequency  distribution  of  standardized  effect  sizes  for  problem-solving 

achievement.  Expected  nonnal  distribution  curve  is  illustrated.  Note.  Data 
are  derived  from  ten  independent  studies  involving  1,236  participants. 
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Examination  of  Select  Included  Studies 

Schjtiidt  et  al.  (1996)  compared  the  clinical  problem-solving  skills  of  612  students 
m  five  academic  years  (pre-clinical  years  two,  three  and  four  and  clinical  years  five  and 
six)  at  Dutch  medical  schools  with  lecture-based  (two  schools)  and  problem-based  (one 
school)  cun-icula.  Problem-based  instruction  is  operational ized  as  reiterative,  with  expert 
tutored  groups  in  which  student-generated  learning  issues  guide  individual  and  shared 
learning  assessed  via  triple  jump  and  progress-testing  methods  as  discussed  in  Chapter  2. 
One  lecture-based  school  operated  a  traditional,  teacher-centered,  discipline-oriented 
curriculum  and  the  other  provided  an  integrated  approach  (didactics  integrating 
biomedical  and  clinical  science).  Although  not  randomly  assigned  to  a  particular 
curriculum,  in  the  Netherlands  students  are  assigned  to  specific  medical  schools  via  a 
national  lottery  (i.e.,  quasi-randomized).  Problem-solving  assessment  incorporated  30- 
diagnostic  problems  epidemiologically  representative  of  Dutch  medical  practice.  There 
were  no  differences  in  problem  solving  (diagnostic)  accuracy  between  students  from  the 
integrated  and  PBL  curricula.  However,  students  undergoing  discipline-based,  lecture- 
based  learning  scored  significantly  lower  than  those  from  the  integrated  and  PBL 
curricula.  The  integrated  instructional  program  was  lecture-based,  but  did  not  meet 
operational  characterisfics  for  this  meta-analysis.  No  other  comparative  investigation  in 
the  literature  included  lecture-based  integration  of  biomedical  and  clinical  science. 
Therefore,  only  traditional  discipline-based,  LBL  and  PBL  curricular  outcomes  were 
compared.  Differential  instructional  scores  for  students  in  pre-clinical  educafion  were  not 
significantly  different  and  the  data  were  pooled  for  summary  (ES  =  0.00,  95%CI  -0.44  to 
0.44).  However,  during  clerkship  training,  problem-based  students  exhibited  superior 
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problem-solving  skills  (pooled  ES  =  0.38,  95%CI  0.06  lo  0.69).  Although  an  aggregate 
effect  size  of  0.20  is  not  significant  because  the  95%CI  -0.05  to  0.35  contains  zero,  the 
results  suggest  students  trained  via  integration  of  biomedical  and  clinical  science  in  the 
context  of  clinical  problems  eventually  become  more  accomplished  problem  solvers 
(diagnosticians)  and,  presumably,  better  physicians. 

Discovering  problem-solving  achievement  parity  between  PBL  and  lecture-based, 
integrated  curriculum  is  an  important  finding  with  significant  implications  for 
instructional  and  curricular  structure.  Indeed,  the  authors  suggest  that  the  integration  of 
biomedical  and  clinical  science  grounded  in  patient  problems  is  likely  an  essential  feature 
enabling  superior  clinical  problem  solving  performance  independent  of  whether  teaching 
and  learning  are  teacher-  or  student-centered.  If  the  authors  are  correct  and  an  integrated 
curriculum  and  PBL  curriculum  affect  equivalent  problem-solving  success,  what  about 
other  important  aims  of  medical  education,  for  example,  the  development  of 
collaboration  skills  and  self-directed  learning?  If  achievement  of  content  reproduction 
and  problem  solving  skills  are  the  only  educational  goals,  integrafion  of  biomedical  and 
clinical  science  via  teacher-centered,  lecture-based  instructional  format  may  be 
efficacious.  From  this  vantage,  the  importance  of  non-cognitive  outcomes  and/or 
implementation  expense  likely  would  detennine  the  fate  of  problem-based  teaching  and 
learning.  Importantly,  and  in  a  relatively  robust  manner,  the  results  of  this  data  analysis 
illustrate  differential  cognitive  effects  with  varied  curricular  approaches  in  well-matched 
students.  Moreover,  sensitivity  to  distinguish  differenfial  instructional  effects  renders  the 
instrument  worthy  of  further  study.  The  instrument  could  be  adapted  to  reflect 
geographically  relevant  epidemiologically  confimicd  illness  and  wellness  patterns. 
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Results  of  this  analysis  also  emphasize  the  importance  of  assessing  developmental 
patterns  to  examine  instructional  impact.  Because  measuring  differential  instructional 
effects  at  specific  times  may  underestimate  or  overestimate  instructional  effectiveness 
relative  to  the  broader  context  of  academic  development.  That  is,  differential 
instructional  effects  might  just  be  a  matter  of  timing. 

Hmelo.  Gotterer,  and  Bransford  ( 1 997)  studied  forty  second  year  medical  students 
at  Vanderbilt  University  who  completed  different  elective  courses  using  problem-based 
(n  =  20)  or  lecture-based  (n  =  20)  instruction.  Problem-based  instruction  involved  tutor- 
facilitated  collaboration  in  groups  of  eight  for  two  hours  once  a  week  to  resolve  posed 
problems.  Students  were  encouraged  to  formulate  hypotheses  and  generate  learning 
issues.  Learning  issues  regarded  relevant  by  the  group  were  partitioned  among  members 
to  be  resolved  employing  material  (e.g.,  textbooks,  journals,  computer  searches)  and 
human  (e.g.,  experts)  information  resources.  At  subsequent  groupings,  students  shared 
information  vis-a-vis  learning  issues  as  they  attempted  to  resolve  the  problem.  In  each 
group,  50%  of  students  were  either  in  the  first  three-months  of  their  freshman  year  or  first 
month  of  their  sophomore  year.  All  students  were  undergoing  the  same  core  lecture- 
based  curriculum.  Students  were  paid  for  participation.  Problem-solving  skills  were 
assessed  after  students  were  given  information  regarding  the  complaint,  history,  physical 
examination,  and  laboratory  data  in  a  pediatric  patient  with  diabetes.  Students  were 
asked  to  write  pathophysiological  explanations  to  account  for  each  element  of 
information.  The  responses  were  coded  according  to  (a)  use  of  biomedical  science  (i.e., 
pathophysiological  concepts)  and  (b)  reasoning  coherence  was  operationalized  as  the 
maximum  number  ofrelational  operators  linked  in  a  pathophysiological  explanation  (i.e.. 
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length  of  reasoning  chains).  Data  were  analyzed  using  a  two-factor  analysis  of  variance. 
The  data  were  presented  as  pooled  because  there  were  no  interactions  between  student 
academic  level  and  either  use  of  biomedical  science  concepts  or  reasoning  coherence 
outcomes.  The  authors  observed  in  PBL  students  statistically  equivalent  use  of  scientific 
concepts  (F  =  3.09,  dr=  1/36,  p  <  .10;  ES  =  0.57,  95%CI  -0.06  to  1.20)  and  greater 
reasoning  coherence  (F  =  7.22,  df  =  1/36,  p  <  .01 ;  ES  =  0.87,  95%CI  0.22  to  1 .52). 
Effect  size  estimates  were  computed  from  F  statistic  and  sample  sizes  using  the  methods 
described  in  Chapter  3.  These  results  suggest  cognitive  measures  can  be  employed  to 
differentiate  students'  problem-solving  strategies  despite  transient  exposure  to  problem- 
based  teaching  and  learning.  There  are  at  least  two  plausible  rival  explanations  for  these 
results.  The  first  is  selection  bias.  Students  who  elected  to  take  a  course  employing 
problem-based  instruction  may  have  exhibited  observed  problem-solving  skills  a pnon. 
Second,  only  one  case  was  employed  to  assess  problem-solving  skills.  Thus,  the  results 
may  reflect  specific  cases  rather  than  be  due  to  a  differential  instructional  structure 
(Elstein  et  al.,  1978).  Elstein  and  colleagues  argue  medical  problem  solving  is  not 
generalizable.  Rather  clinical  problem  solving  expertise  is  proportional  to  domain 
knowledge  in  general  and  situational  experience  in  particular,  that  is,  case-specific 
knowledge  or  "illness-scripts"  as  discussed  in  Chapter  2.  From  this  perspective,  expertise 
IS  problem-  or  case-specific.  Indeed,  recent  research  suggests  clinical  problem-solving 
success  is  primarily  infiuenced  by  the  typicality  of  case  presentation  (Papa,  Stone,  & 
Aldnch,  1996).  If  correct,  educators  must  carefully  select  and  focus  clinical  problems 
consistent  with  the  objectives  of  instruction.  More  light  might  be  shed  on  these  issues  by 
conducfing  longitudinal  investigations  in  subjects  with  baseline  equivalence  challenged 
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with  varied  clinical  problems.  An  appropriate  reservoir  ofhealth  care  problems  might 
best  be  derived  from  local  epidemiologic  data,  as  suggested  in  the  study  by  Schmidt  et  al. 
(1996)  described  earlier. 

Kaufriian  and  Mann  (1998)  compared  problem-solving  skills  in  three  consecutive 
senior  classes  of  medical  students  at  Dalhousie  University  (Halifax,  Nova  Scotia).  One 
class  (1995)  matriculated  when  pre-clinical  curriculum  (first  two  years)  was  lecture- 
based,  and  two  classes  (1996,  1997)  matriculated  after  pre-clinical  curriculum  conversion 
to  problem-based  learning.  Analyses  of  variance  revealed  no  intergroup  differences  in 
academic  measures  employed  for  admission  or  medical  school  grade  point  average. 
Problem-solving  skills  were  measured  using  Medical  Council  of  Canada  Qualifying 
Examination  (MCCQE)  Part  I.  Part  I  examines  the  competence  of  MD  degree  recipients 
for  conducting  supervised  clinical  practice  in  residency  training.  The  problem-solving 
element  of  the  examination  is  a  timed  (four  hours)  multi-method  assessment 
incorporating  short-menu  (up  to  40  options)  and  short-answer  write-in  questions. 
Clinical  problems  are  presented,  followed  by  one  or  more  questions  designed  to  assess 
key  features  in  the  solution.  Specifically,  examinees  are  asked  to  elicit  clinical  data, 
prescribe  diagnostic  procedures,  render  diagnoses  and  prescribe  treatment.  Reliability 
coefficients  for  problem-solving  items  in  1996  were  KR-20  =  .90  and  alpha  =  .71  and  in 
1997  were  KR-20  =  .90  and  alpha  =  .72.  Problem-based  learning  students  scored  higher 
in  1 996  (M  =  5 1 5.4,  SD  =  8 1 .0)  and  1 997  (M  =  530.2,  SD  =  66.2)  compared  to  LBL 
students  (M  =  499.0,  SD  =  75.2)  with  effect  size  estimates  of  .21  (95%C1  -0.10  to  0.52) 
and  0.44  (95%C1  0.12  to  0.76),  respectively.  The  study  effect  size  estimate  of  0.33  is 
significant  because  the  95%C1  0.02  to  0.64  does  not  include  zero. 
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In  a  follow-up  study,  Kaufman  and  Mann  (1999)  examined  the  differential 
curricular  effect  in  problem-solving  skills  assessed  by  MCCQE  Part  II  in  graduates  of 
PBL  (n  =  72)  and  LBL  (n  =  68)  curricula  at  Dalhousie  University.  Part  II  examines  the 
competence  of  candidates  after  at  least  1 8-months  of  residency  and  before  independent 
clinical  practice.  Part  II  assesses  problem-solving  skills  in  Medicine,  Pediatrics, 
Obstetrics  and  Gynecology,  Preventive  Medicine  and  Community  Health,  Psychiatry,  and 
Surgery.  The  examination  format  incorporates  clinical  stations  where  candidates  interact 
with  a  standardized  patient.  Baseline  academic  equivalence  of  groups  was  confinned. 
Problem-solving  scores  were  statistically  equivalent  for  the  1996  PBL  class  (M  =  512.7, 
SD  =  88.1)  and  LBL  (1995)  class  (M  =  51 1.7,  SD  =  82.3)  with  an  estimated  effect  size 
estimate  of  0.01  (95%CI  -0.32  to  0.34).  Unfortunately,  this  is  the  only  study  reporting 
the  relative  effects  of  LBL  and  PBL  regarding  the  problem-solving  component  of 
MCCQE  Part  II. 

Self-Directed  Learning  Skills 

Hypothesis  7.  Aggregated  effect  size  estimates  for  self-directed  learning  skills  in 
students  and  physicians  who  undergo  medical  education  with  problem-based  versus 
lecture-based  teaching  and  learning  will  be  significantly  positive  because  the  lower  limit 
of  95%  confidence  interval  will  exceed  zero.  Measures  of  self-directed  learning  were 
derived  from  a  variety  of  sources  including  specific  sub-scales  of  Mitchell's  Cognitive 
Behavior  Survey  (e.g.,  memorization,  conceptualization,  reflection,  peer  interaction)  and 
the  Medical  School  Learning  Environment  Survey  (e.g.,  meaningful  learning),  library 
databases,  locally  constructed  surveys  to  access  study  and  learning  patterns  and  learning 
resource  utilization,  and  current  practice  competence.  Reliability  and  construct  validity 
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have  been  reported  for  Mitchell's  Cognitive  Behavior  Survey  (Mitchell,  1994)  and 
Medical  School  Learning  Environment  Survey  (Feletti  &  Clark  1981a,  1981b). 
Standardized  effect  size  estimates  and  associated  95%  confidence  intervals  computed 
from  self-directed  learning  outcomes  presented  in  nineteen  studies  investigating  3,487 
participants  are  summarized  graphically  in  Figure  4-9.  Inspecting  the  plot  of  95% 
confidence  intervals  for  each  effect  size  reveals  that  eleven  confidence  intervals  are 
above  zero,  none  are  below  zero,  and  eight  include  zero.  The  mean  effect  size  estimate  is 
included  in  twelve  of  nineteen  confidence  intervals.  Students  and  physicians  undergoing 
problem-based  (n  =  1,116)  versus  lecture-based  (n  =  2,371)  learning  exhibited 
significantly  greater  self-directed  learning  skills,  ES  =  0.47,  95%CI  0.39  to  0.55. 
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Figure  4-9.  Standardized  effect  sizes  and  associated  95%  confidence  intervals  for  self- 
directed  learning  outcomes.  Note.  Data  are  derived  from  1 9  independent 
studies  involving  3,487  participants.  Study  number  87  appears  twice  because 
two  independent  samples  were  reported,  as  detailed  in  text.  Citation  for 
study  number  is  listed  in  Appendix  A.  Dashed  line  represents  mean  weighted 
effect  size. 
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This  finding  suggests  problem-  based  instruction  under  average  conditions  would  be 
expected  to  transition  self-direeted  learning  skills  in  a  typical  learner  in  a  lecture-based 
instruction  population  from  the  50"'to  the  68"'  percentile.  Publication  bias  for  self- 
directed  learning  outcomes  appears  unlikely  because  there  was  no  significant  correlation 
between  effect  size  estimates  and  sample  size  (Speamian's  rho  =  -.05,  g  =  .83). 
According  to  Rosenthal's  (1979)  method,  1,013  additional  studies  with  a  mean  effect  size 
of  zero  are  required  to  reduce  the  combined  significance  of  self-directed  learning 
outcomes  to  less  than  .05. 

Hypothesis  8.  Variation  in  self  directed  learning  effect  size  estimates  will  exceed 
random  error  as  evidenced  by  a  Q-statistic  at  the  .05  level  of  significance  warranting 
subgroup  analyses  to  test  for  potential  moderation  by  a  priori  defined  outcome  measures, 
methodological  features,  and  substantive  features  of  problem-based  teaching  and 
learning.  The  frequency  distribution  of  standardized  effect  size  estimates  for  self- 
directed  learning  skills  are  represented  in  Figure  4-10.  The  result  of  homogeneity 
analysis  suggests  the  distribution  of  effect  size  estimates  for  self  directed  learning  are 
greater  than  would  be  expected  entirely  from  sampling  error,  Q(  1 8)  =1 1 1 .8,  p  <  .001 . 
This  finding  suggests  further  exploration  is  warranted  to  discover  whether  study  features 
are  operating  to  moderate  variance  of  effect  sizes.  The  distribution  of  within-group  and 
between-group  variances  of  effect  size  estimates  associated  with  outcome  features  are 
summarized  in  Table  4-10. 

Outcome  measurement  features.  Self-directed  learning  effect  size  estimates  are 
summarized  in  Table  4-10.   Effect  size  estimates  derived  from  varied  measures  of  self- 
directed  learning  were  statistically  equivalent,  Q„(3)  =  3.2,  p  =  .36.  Students  undergoing 
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Figure  4-10.  Frequency  distribution  of  standardized  effect  sizes  for  self-directed  learning 
outcomes.  Expected  normal  distribution  curve  is  illustrated.  Note.  Data 
are  derived  from  19  independent  studies  involving  3,487  participants. 

Table  4-10.  Mean  effect  sizes  and  variance  distribution  for  self-directed  learning  as  a 
function  of  outcome  measures 


N 

ES 

95%CI 

Homogeneity 

Variables 

Lower 

Upper 

Qb        Qwi 

Outcome  Type 

3.2 

Resource  utilization 

5 

.40 

.25 

.54 

18.0 

Study  time 

8 

.51 

.33 

.69 

77.7 

Learning  process 

4 

.46 

.33 

.59 

12.8 

Practice  competence 

2 

.67 

.39 

.94 

0.2 

Note.  N  =  number  of  weighted  effect  size  (ES)  estimates  averaged.  If  95%  confidence 
mterval  (CI)  does  not  contain  zero,  ES  is  significant  at  the  .05  level.  Qb  =  between-group 
variance;  Qwi  =  within-group  variance  for  each  feature  sub-group. 

problem-based  versus  lecture-based  learning  manifested  significantly  greater  self- 
directed  learning  skills  regardless  of  outcome  measure  type.  The  magnitude  of  problem- 
based  instructional  effects  was  relatively  large  when  measures  assessed  practice 
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competence  (i.e.,  remaining  current  in  a  clinical  practice  domain),  ES  =  0.67,  but  only 
two  studies  contributed  data.  Thus,  effect  size  for  self-directed  learning  appears  to  be 
explained  by  outcome  measure  and  analyses  of  methodological  and  substantive  features 
as  potential  modifiers  of  effect  size  magnitude  is  unwaiTanted  (Hedges,  1994). 
Examination  of  Select  Included  Studies 

In  a  longitudinal  study,  Newble  and  Clarke  ( 1 986)  revealed  differential 
mstructional  effects  on  self-directed  learning  approaches  in  Australian  medical  students 
undergoing  problem-based  versus  lecture-based  instruction.  In  Australia,  biomedical 
science  and  clinical  science  are  taught  during  the  first  three  years  and  the  last  three  years 
of  medical  school,  respectively.  First,  third  and  final-year  students  undergoing  LBL 
(University  of  Adelaide)  and  PBL  (University  of  Newcastle)  were  invited  to  complete  a 
64-item  survey  (Lancaster  Approaches  to  Studying  Inventory)  after  completing 
approximately  two-thirds  of  the  academic  year.  Survey  items  are  grouped  into 
dimensions  including  meaning  orientation  (deep  approach,  interrelated  ideas,  use  of 
evidence,  intrinsic  mofivation)  and  achieving  orientation  (strategic  approach, 
disorganized  study  methods,  negative  attitudes  toward  studying,  achievement 
motivation).  Eighty-seven  percent  (n  =  97),  89%  (n  =  94),  and  91%  (n  =  43)  of  first, 
third  and  final-year  LBL  students  completed  the  survey.  Ninety-five  percent      (n  =  63), 
81%  (n  =  46)  and  76%  (n  =  44)  of  first,  third  and  final-year  PBL  students  completed  the 
inventory.  The  authors  did  not  present  raw  means  and  standard  deviations,  rather  they 
presented  t  statistics  and  exact  p  values  for  each  contrast.  Standardized  effect  size 
estimates  were  computed  using  the  methods  described  in  Chapter  3.  Meaning  orientation 
effect  size  estimates  were  significantly  greater  in  first  year  PBL  than  LBL  students  (ES  = 


0.73,  95%CI  0.40  to  1 .06)  and  second  year  (ES  =  0.72,  95%CI  0.39  to  1 .05),  but  not  final 
year  (ES  =  0.29,  95%CI  -0.13  to  0.71).  Achieving  orientation  scores  were  significantly 
greater  in  PBL  than  LBL  students  in  first  year  (ES  =  0.67,  95%CI  0.34  to  1 .00)  and  final 
year  (ES  =  0.61 ,  95%CI  0. 1 8  to  1 .04),  but  inexplicably  not  in  second  year  (ES  =  0.24, 
95%CI  -0.08  to  0.56).  A  limitation  of  this  study  is  the  lack  of  data  to  assess  baseline 
equivalence  of  groups.  Thus,  selection  bias  is  a  plausible  rival  explanation  of  the  results. 
However,  examination  of  sub-scales  within  major  dimensions  indicates  important 
developmental  changes  associated  with  problem-based  teaching  and  learning.  For 
example,  the  deep  processing  dimension  increases  from  an  effect  size  estimate  of  0.33 
(95%CI  0.01  to  0.65)  in  first  year  to  0.53  (95%CI  0.21  to  0.85)  in  year  three. 

Blumberg  and  Michael  (1992)  compared  patterns  of  self-directed  learning  in  PBL 
and  LBL  students  at  Rush  Medical  College  using  data  quantifying  student  self  reported 
learning  resource  use,  library  circulation  archives,  student-reported  perceptions  of  their 
proficiency  in  specific  self  directed  learning  skills,  and  faculty-reported  perceptions 
about  students'  self  directed  learning  skills.  Effect  size  estimates  were  computed  for 
survey  data  (ES  =  0.38,  95%CI  -0.02  to  0.82),  library  resource  data  (ES  =  0.79,  95%CI 
0.38  to  1 .20),  and  the  study  (ES  =  0.58,  95%C1  0. 1 8  to  0.98).  PBL  students  tended  to 
employ  textbooks,  journals,  and  informal  discussions  while  LBL  students  employed 
course  syllabi  and  verbatim  transcriptions  of  lectures.  Students  undergoing  problem- 
based  instruction  were  more  likely  to  report  self  selecting  material  to  study,  while  LBL 
students  identified  reading  specific  teacher-generated  assignments.  Library  circulation 
data  confirmed  PBL  students  borrowed  more  materials  than  did  LBL  students.  However, 
library  data  were  not  stratified  by  resource  borrowed,  nor  did  data  account  for 
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infonnation  acquired  by  means  other  than  borrowing  resources,  for  example,  photocopied 
mfonnation  or  information  obtained  via  consulting  resources  at  the  library,  but  not 
borrowed.  The  results  suggest  PBL  students  perceived  they  are  more  self-directed 
learners.  Faculty  reports  substantiated  student  perceptions.  Faculty  observed  that  PBL 
compared  to  LBL  students  were  more  likely  to  admit  and  subsequently  pursue  their 
knowledge  gap.  Moreover,  faculty  reported  PBL  students  were  more  efficient  at  finding 
relevant  references  and  at  obtaining  necessary  information.  Interpretation  is  hampered 
because  the  authors  do  not  assess  the  baseline  equivalence  of  LBL  and  PBL  students. 
Thus,  cross-sectional  differences  attributed  to  instruction  might  have  existed  a  priori.  In 
addition,  results  are  limited  by  self-reporting;  but  they  are  indirectly  corroborated  using 
library  circulation  databases  and  fiiculty  observations. 

Shin  and  colleagues  (1993)  sent  to  a  random  sample  of  family  or  general  practice 
physicians  who  graduated  from  McMaster  University  (PBL  pre-clinical  curriculum)  or 
University  of  Toronto  (LBL  pre-clinical  cun-iculum)  an  analytic  survey  to  compare 
cun-ent  knowledge  about  management  and  treatment  of  hypertension.  A  self- 
administered  hypertension  knowledge  questionnaire  previously  was  validated  and 
revealed  an  inverse  relationship  with  time  since  graduation  from  medical  school 
involving  three  different  communities  in  two  previous  studies  including  -0.55  (p  <  .001 ) 
in  one  study  (Evans  et  al..  1984)  and  -0.63  (p  <  .001)  in  the  control  group  in  the  other 
investigation  (Evans  et  al.,  1986).   Moreover,  in  a  randomized  comrolled  investigation, 
performance  on  a  hypertension  knowledge  questionnaire  was  significantly  increased  in 
response  to  an  educational  intervention  (Evans  et  al.),  thus,  confirming  the  instructional 
sensitivity  of  the  instrument  (Crocker  &  Algina,  1986).  The  questionnaire  is  changed 
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only  to  update  information  according  to  the  guidelines  of  the  Canadian  Hypertension 
Society.  The  questionnaire  comprised  five  areas  of  practice  appropriate  for  hypertension 
management.  These  included  methods  of  measuring  blood  pressure  (15  points), 
recommended  blood  pressures  for  treatment  (10  points),  phannacologic  treatments  (26 
points),  nonpharmacologic  treatment  (5  points),  and  patient  follow-up  (26  points), 
including  the  frequency  of  return  appointments,  actions  taken  after  blood  pressure 
normalizes,  and  detection  and  management  of  patient  noncompliance.  Sample  size  was 
designed  to  detect  a  10%  difference  in  mean  scores  with  a  standard  deviation  of  13% 
using  a  two-tailed  type  I  error  rate  of  .05  and  power  of  .9  allowing  for  a  non-response  rate 
of  20%.  Graduates  of  PBL  curriculums  (M  =  68%)  scored  higher  than  those  from  LBL 
curriculums  (M  =  62%).  The  authors  presented  the  results  of  the  t  test  (p  <  .001),  but 
computed  effect  size  of  0.59  also  is  significant  because  a  95%  confidence  interval  from 
0. 1 5  to  1 .03  does  not  contain  zero.  Importantly,  the  authors  employed  a  multivariate 
linear  regression  to  assess  potential  moderating  variables  including  number  of  years  since 
graduation,  number  of  years  in  practice,  medical  school  of  graduation,  certification  from 
appropriate  medical  specialty  board,  gender,  number  of  patients  seen  per  working  day, 
and  proportion  of  practice  involving  patients  with  hypertension.  Analyses  revealed  the 
only  factor  predicting  hypertension  knowledge  score  was  curriculum  type  (g  <  .01).  The 
authors  did  not  present  correlation  coefficients,  but  slope  of  line  regressing  years  of 
practice  on  hypertension  knowledge  scores  was  .330  and  .002  for  lecture-based  and 
problem-based  learning  graduates,  respectively.  Importantly,  these  data  suggest  PBL 
graduates  sustain  practice  competence  for  state-of-the-art  hypertension  knowledge  and 
treatment  regimens  virtually  independent  of  practice  experience  (five  to  16  years). 
Indeed,  the  most  important  outcome  of  SDL  is  practice  patterns  exhibiting  self-direction. 
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Richards  and  Cariaga  (1993)  observed  differential  study  patterns  for  licensing 
examinations  in  academically  matched  second  year  medical  students  undergoing  PBL 
versus  LBL  at  Wake  Forrest  University.  Employing  a  retrospective,  self-report,  students 
were  asked  to  estimate  hours  per  week  spent  preparing  for  the  NBME  Part  I  during  their 
two  preclinical  years.  The  NBME  Part  I  examination  is  taken  by  students  near  the  end  of 
their  second  year  of  medical  school.  Students  were  asked  to  partition  their  response  into 
Fall,  Winter,  and  Summer  teniis  of  their  first  year,  and  Fall,  Winter  and  after  completion 
of  classes  in  the  second  year.  Eighty-two  percent  and  79  percent  of  students  undergoing 
problem-based  and  lecture-based  curricula,  respectively,  completed  the  survey. 
Cumulative  study  time  was  greater  in  PBL  students  (M  =  786  hours)  versus  LBL  students 
(M  =  416  hours)  (g  <  .01).  There  were  no  differences  in  study  patterns  during  the  first 
year  of  medical  school.  Significant  differentials  in  per  week  study  patterns  between  PBL 
versus  LBL  were  observed  during  Fall  (M  =  4.8  hours  vs.  M  =  2. 1  hours,  g  <  .05),  Winter 
(M  =  1 7.5  hours  vs.  M  =  7.2  hours,  p  <  .05),  but  not  in  the  few  weeks  immediately  before 
examination  (M  =  50.3  hours  vs.  M  =  65. 1  hours)  at  end  of  second  year.  Interestingly, 
there  was  no  difference  in  examination  scores  between  PBL  (M  =  212)  versus  LBL  (M  = 
209)  (g  =  .45).  Because  NBME  Part  1  is  designed  to  assess  biomedical  science 
knowledge  employing  a  multiple  choice  question  fomiat,  the  assessment  is  compatible 
with  the  teaching  and  learning  characteristic  of  lecture-based  instruction,  but  incongruent 
with  those  of  problem-based  instruction.  Students  undergoing  PBL  apparently 
compensate  for  their  curriculum  by  studying  longer.  If  correct,  the  "compensation 
hypothesis"  advanced  by  the  authors  has  significant  educational  implications.  Although 
these  data  confimi  medical  students'  ability  to  adapt,  there  is  an  undesirable  diversion  of 
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energy  and  time  away  from  the  teaching  and  learning  objectives  of  PBL.  As  discussed  in 
Chapter  2,  since  licensing  examinations  are  permanent  elements  of  the  academic 
landscape  (i.e.,  "hidden  curriculum,"  or  other-directed  learning),  a  challenge  for  problem- 
based  instmction  designers  is  to  blunt,  or  obviate  a  student's  need  to  compensate  for  a 
perceived  lack  of  factual  knowledge,  while  facilitating  license  exam  success. 
Zimmerman  and  Lebeau  (2000)  suggest  that  in  addition  to  learning  objectives  for 
particular  content  areas,  linking  problem-specific  objectives  to  licensing  examinations, 
"reinforces  this  other-directed  inHuence  on  self-directed  generation  of  learning  issues"  (p. 
304).  Indeed,  the  impending  high  stakes  of  assessments  potentially  can  be  converted  to 
"teachable  moments." 

Hill,  Rolfe,  Pearson,  and  Heathcore  (1998)  studied  graduates  from  LBL  and  PBL 
medical  schools  during  their  first  year  of  postgraduate  training  (e.g.,  internship)  in  New 
South  Wales,  Australia.  All  graduates  of  PBL  (University  of  Newcastle)  and  a  random 
sample  of  graduates  from  the  two  LBL  medical  schools  (University  of  New  South  Wales 
and  University  of  Sydney)  were  surveyed  with  the  Preparation  for  Hospital  Practice 
Questionnaire.  This  sur\'ey  instrument  included  a  self-directed  learning  sub-scale  with 
six-Items,  which  sought  to  examine  self-evaluation  ofclinical  performance,  self- 
identification  of  educational  need,  and  self-extension  of  knowledge  and  skills.  Data  were 
analyzed  from  139  questionnaires  representing  a  response  rate  of  60%  including  78%  (n 
=  52)  and  52%  (n  =  87)  from  PBL  and  LBL  medical  school  graduates,  respectively. 
Reliability  of  the  self-directed  learning  sub-scale  was  .82  (Cronbach's  alpha).  Goodness- 
of-fit  analyses  were  performed  to  examine  the  representativeness  of  the  sample  (e.g., 
gender).  Analysis  of  variance  was  used  to  examine  differences  in  age  of  responders  and 
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non-responders.  Covariates  of  age  and  gender  were  not  significant  factors  for  self- 
directed  learning  sub-scale  data.  1  pooled  questionnaire  item  data  and  computed  an  effect 
size  of  0.71  which  is  significant  because  95%CI  0.36  to  1.06  does  not  include  zero.  On 
average,  PBL  graduates  indicated  they  were  better  prepared  to  manage  their  own 
learning.  These  findings  provide  evidence  that  self-directed  learning  strategies  associated 
with  problem-based  instruction  lingers  and  affects  perception  of  practice  preparation  and 
confidence  in  physicians  during  their  first  postgraduate  year  of  training. 

Ozuah,  Curtis  and  Stein  (2001 )  investigated  the  impact  of  problem-based 
instruction  on  self-directed  learning  in  pediatric  residents.  After  three  months  of  daily 
lecture-based  instruction,  residents  were  assigned  to  either  twice-weekly  problem-based 
instruction  (n  =  39)  or  continued  lecture-based  instruction  (n  =  41 ).  Residents'  clinical 
rotation  schedule  was  used  for  assignment  to  problem-based  or  lecture-based  instruction. 
Problem-based  instruction  was  operationalized  according  to  the  reiterative  model 
(Barrows,  1986).  Specifically,  unprepared  learners  in  small  groups  are  presented  with  a 
clinical  problem.  Faculty  tutors  guide  collaboration  without  providing  factual 
knowledge.  Group-generated  learning  issues  are  self-assigned  for  inquiry.  Two  to  three 
days  later,  findings  ofself-directed  inquiry  are  shared  among  group  members.  Closing 
the  learning-loop,  sessions  culminate  with  evaluation  and  synthesis  of  the  learning 
process  including  discussion  of  errors  emerging  during  the  process.  Residents  completed 
four  problems  per  month.  Three  months  later,  problem-based  instruction  was 
discontinued  and  all  residents  converged  to  lecture-based  instruction  for  three  months. 
Residents  were  given  questionnaires  during  the  initial  lecture-based  period  (baseline)  and 
monthly  throughout  the  study  to  evaluate  self-directed  learning  patterns,  including  time 
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committed  to  independent  study  (hours/weel<),  medical  discussions  (frequency/week)  and 
computer  literature  searches  (frequency/week).  There  were  no  intergroup  differences  in 
study  time,  medical  discussions,  or  computer  searches  at  baseline.  Baseline  self-directed 
learning  behaviors  by  residents  undergoing  lecture-based  instruction  were  sustained 
throughout  the  study.  During  intervention,  a  greater  percentage  of  residents  undergoing 
problem-based  compared  to  lecture-based  instruction  reported  study  time  of  more  than  5 
hours  per  week  (26%  vs.  7%.  p  =  .001 ).  more  than  5  hours  per  week  of  medical 
discussions  (28%  vs.  4%,  p  =  .005).  and  two  or  more  computer  searches  conducted  per 
week  (51%  vs.  30%,  p  =  005).  However,  self-directed  learning  patterns  in  residents  who 
underwent  problem-based  instruction  returned  to  baseline  after  reverting  to  lecture-based 
instruction.  According  to  the  methods  described  in  Chapter  3,  proportional  differences 
for  each  self-directed  learning  measure  were  transformed  to  effect  size  estimates  and 
pooled  for  summary,  ES  =  0.55,  95%CI  0.10  to  1.00.  Although  selection  bias  threatens 
mtemal  validity,  the  baseline  equivalence  is  supported  by  intergroup  parity  in  pre- 
imervention  self-directed  learning  measures.  Self-reported  data  also  threaten  validity,  but 
the  authors  report  residents  were  blinded  to  research  hypothesis  and  approximately  30% 
of  residents,  independent  of  instructional  format,  consistently  reported  studying  one  hour 
or  less  per  week.  This  observation  suggests,  "residents  were  very  willing  to  report  low 
levels  of  self-directed  learning  behaviors"  (p.  671).  Unfortunately,  whether  increased 
self-directed  learning  was  directly  related  to  learning  issues  generated  by  problem-based 
tutorial  sessions  or  to  supplement  these  "assignments"  is  unclear.  In  other  words,  was 
self-directed  learning  exclusively  motivated  to  attain  specific  information,  and  no  more? 
Although  learning  issues  are  student-group  generated,  they  functionally  are  assignments. 
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If  behaviors  indicating  increased  study  time,  medical  discussions,  and  computer  searches 
are  driven  by  problem-based  assignments,  then  it  is  no  surprise  this  behavior  vanished 
post  intervention,  hi  this  context,  independent  learning  is  inspired  by  problem-based 
instruction.  However,  once  problem-based  instruction  ceases,  so  does  increased  self- 
directed  learning  behavior  as  evidenced  by  the  return  to  baseline  level.  Whether  longer 
exposure  to  problem-based  instrtiction  would  foster  a  persistent  disposition  toward  self- 
directed  learning  in  resident  physicians  is  unclear,  but  an  important  question  begging 
research. 
Examination  of  Select  Excluded  Study 

Coles  (1985)  conducted  a  longitudinal  investigation  to  compare  approaches  to 
self-directed  learning  in  medical  students  undergoing  problem-based  and  lecture-based 
instruction  at  two  different  schools  in  England.  Unfortunately,  the  author  did  not  report 
the  nuinber  of  subjects,  thus  a  weighted  standardized  effect  size  could  not  be  computed. 
A  survey  was  completed  by  students  at  entry  and  completion  of  the  first  year  of  medical 
school,  hiitial  and  repeat  response  rates  were  94%  and  75%  for  LBL  students  and  62% 
and  52%  for  PBL  students,  respectively.  Dimensions  of  self-directed  learning  were 
operationalized  as  reproducing,  meaning,  and  versatility.  Longitudinal  differences  in 
students  undergoing  LBL  included  a  significant  increase  in  reproduction  (p  <  .05)  and 
decreases  in  meaning  (p  <  .001 )  and  versatility  (p  <  .001 ).  Obser\'ed  longitudinal  affects 
of  PBL  included  reduction  in  reproducing  (p  <  .01 )  and  no  changes  in  other  approaches  to 
learning  dimensions.  Differential  learning  effects  of  instructional  format  emerged  at  the 
end  of  first  year  as  evidenced  by  significantly  lower  reproducing  (g  <  .01 )  and  greater 
meaning  (p  <  .001 )  and  versatility  (p  <  .01 ).  The  results  indicate  problem-based  teaching 
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and  learning  cultivates  desirable  approaches  to  self-directed  learning,  while  lecture-based 
instruction  facilitates  the  adaptation  of  undesirable  study  patterns.  Although  data  were 
derived  from  self-reports,  baseline  equivalence  and  longitudinal  design  sponsor  validity 
of  this  study.  Further  confirmation  of  undesirable  study  patterns  in  medical  students 
induced  by  lecture-based  instruction  is  provided  by  Lindblom-Ylanne  and  Lonka  (1999, 
2002)  and  was  discussed  in  Chapter  2. 

Attitude  Toward  Learning 
Hypotliesis  9.  Aggregated  effect  size  estimates  for  positive  attitudes  toward 
learning  in  students  and  physicians  who  undergo  medical  education  with  problem-based 
versus  lecture-based  teaching  and  learning  will  be  significantly  positive  because  the 
lower  limit  of  95%  confidence  interval  will  exceed  zero.  Measures  of  attitude  toward 
learning  were  obtained  using  various  instruments  including  the  Medical  School  Learning 
Environment  Survey  (MSLES  Form  A,  Fonn  B),  Course  Experience  Questionnaire 
(CEQ),  Mitchell's  Cognitive  Behavior  Survey  (CBS),  Kellner's  Symptom  Questionnaire, 
Learning  Environment  Questionnaire  (LEQ)  and  investigator  constructed  survey 
instruments.  Reliability  and  construct  validity  have  been  reported  for  the  MSLES  (Feletti 
&  Clark,  1981a,  1981b)  and  CBS  (Mitchell,  1994).  Effect  size  estimates  and  associated 
95%  confidence  intervals  computed  from  attitude  toward  learning  outcomes  presented  in 
thirty  studies  investigating  5,670  subjects  are  summarized  graphically  in  Figure  4-11. 
Examining  the  plot  of  95%  confidence  intervals  for  each  effect  size  reveals  eighteen 
confidence  intervals  are  above  zero,  none  are  below  zero,  and  twelve  include  zero.  Mean 
effect  size  is  included  in  nineteen  of  thirty  confidence  intervals.  Medical  students  and 
physicians  undergoing  PBL  (n  =  1,945)  versus  LBL  (n  =  3,725)  exhibited  significantly 
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more  positive  attitudes  toward  learning,  ES  =  0.45,  95%CI  0.39  to  0.52. 
Inclusion  of  outlier  (ES  =  1.76,  n  =  276)  would  increase  the  mean  effect  size  to  0.49 
(95%C1  0.43  to  0.55),  which  is  not  a  significant  change.  Non-outlier  mean  effect  size 
suggests  problem-based  instruction  is  expected  to  improve  attitude  toward  learning  of 
average  pupil  undergoing  lecture-based  instruction  from  50"'  percentile  to  67"'  percentile. 
Publication  bias  for  attitude  toward  learning  outcomes  appears  unlikely  because  there 
was  no  significant  correlation  between  effect  size  estimates  and  sample  size  (Spearman's 
rho  =  -.23,  E  =  .22).  According  to  Rosenthal's  (1979)  method,  2,518  additional  studies 
with  a  mean  effect  size  of  zero  are  required  to  reduce  the  combined  significance  of 
attitude  toward  learning  outcomes  to  less  than  .05. 
3.0  ■ 

2.5 
2.0 
1.5 


t3     1.0  - 


0.0 
-0.5 


I     4    6    7     S    12  14  15  16  19  20  23  29  i4  35  39  42  51  53  57  58  59  64  66  67  72  81  82  87  87 
Study  Number 
Figure  4-11.  Plot  of  standardized  effect  sizes  and  associated  95%  confidence  intervals 
for  attitude  toward  learning  outcomes.   Note.  Data  are  derived  from  30 
independent  studies  involving  5,670  participants.  Citation  for  study  number 
is  listed  in  Appendix  A.  Dashed  line  represents  mean  weighted  effect  size. 
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Hypothesis  10.  Variation  in  attitude  toward  learning  effect  size  estimates  will 
exceed  random  error  as  evidenced  by  a  Q-statistic  at  the  .05  level  of  significance 
warranting  sub-group  analyses  to  test  for  potential  moderation  by  a  priori  defined 
outcome  measures,  methodological  features,  and  substantive  features  of  problem-based 
teaching  and  learning.  The  frequency  distribution  of  standardized  effect  size  estimates 
for  attitude  toward  learning  outcomes  is  represented  in  Figure  4-12.  Variance  analysis 
greater  than  would  be  expected  entirely  from  sampling  error,  Q(29)  =138.9,  g  <  .00001). 

g 
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-0.4       -0.2        0.0        0.2 


1.4        1.6 


0.4        0.6        0.8 

Effect  Size 

Figure  4-12.  Frequency  distribution  of  standardized  effect  size  estimates  for  attitude 
toward  learning  outcomes.   Expected  nornial  distribution  curve  is 
illustrated.  Note.  Data  are  derived  from  30  independent  studies  involving 
5,670  participants. 
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This  finding  suggests  further  exploration  is  warranted  to  discover  whether  study  features 

are  operating  to  moderate  variance  of  effect  sizes.  The  distribution  of  within-group  and 

between-group  variances  of  effect  size  estimates  associated  with  outcome, 

methodological,  and  substantive  features  are  summarized  in  Tables  4-11,  4-12  and  4-13. 

Outcome  measurement  features.  Attitude  toward  learning  effect  size  estimates 

did  not  vary  significantly  among  measure  sources,  Qn(2)  =1.1,2  =  -60,  or  type  Q„(2)  = 

4.3,  p  =  .12,  and  are  summarized  in  Table  4-11.  Although  often  considered  an  important 

element  of  research  quality,  including  studies  with  diverse  reliability  and  validity 

considerations  did  not  significantly  influence  effect  size  estimates.  Contribution  to 

within-group  variance  (Qw)  was  39%,  5%  and  56%  in  studies  reporting  both  reliability 

and  validity  of  measures,  reporting  reliability  only,  and  reporting  no  reliability  or  validity 

considerafion. 

Table  4-11.  Mean  effect  size  estimates  and  variance  analyses  for  attitudes  toward 
learning  as  a  function  of  outcome  measure 


95%)CI Homogeneity 

Variables N      ES  Lower  Upper  Qb        Qwi 


Outcome  source  1  | 

Valid  /  reliable  instruments  13  .42  .31  .53  57.5 

Reliable  instruments  4  .51  .20  .83  4  8 

Other  instruments  13  .47  .38  .56  75.6 

Outcome  type  4  3 

Posifive  learning  environment  17  .41  .34  .49  535 

Utility  of  education  10  .53  .40  .67  72  9 

Lower  academic  anxiety 3  .65  .23  .87  3.2 


Note.  N  =  number  of  weighted  effect  size  (ES)  estimates  averaged.  If  95%  confidence 
interval  (CI)  does  not  contain  zero,  ES  is  significant  at  .05  level.  Qb  =  between-group 
variance;  Qw;  =  within-group  variance  for  each  feature. 

Methodological  features.  Variance  in  methodological  features  are  summarized 

m  Table  4-12.  There  were  no  differences  in  effect  sizes  claiming  baseline  equivalence 

(ES  =  .48)  versus  those  in  which  it  was  not  addressed  (ES  =  .40).  Methodological 

features  found  to  moderate  attitude  toward  learning  effect  size  estimates  include  those 
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Table  4-12.  Mean  effect  size  estimates  and  variance  analyses  for  attitudes  toward 
learning  as  a  function  of  methodological  features 


Variables 


95%CI 


N       ES 


Lower 


Sample  design 

Randomized  6  .35  .14 

Intact  groups  24  .47  .40 

Baseline  Equivalence 

Yes  24  .48  .40 

No  0 

Not  Reported  6  .40  .26 

Control  group 

Parallel  curriculum /course        16  .48  .38 

Another  institution  4  .46  .16 

Historical  /  self  10  .43  .33 

Longitudinal  design 

Yes  4  "  .52  .30 

No  26  .44  .37 

Multiple  centers 

Yes  6  .33  .18 

No  24  .50  .42 

Response  rate  (%) 

90  to  100  7  .55  .35 

80  to  89  6  .45  .30 

70  to  79  5  .37  .16 

69  or  less  7  .35  .21 

Not  reported  5  .77  .52 

Country 

United  States  12  .51  .42 

Canada  7  .25  .13 

Australia  3  .41  .22 

Scandinavia  3  .94  .71 

United  Kingdom  2  .40  .23 

Germany  3  .57  .06 

Publication  year 

1980-1989  2  .84  .45 

1990-1999  19  .52  .44 

2000-2002 9  .33  .22 


Upper 


.57 
.54 

.55 

.55 

.59 

.75 
.53 

.74 
.51 

.49 

.57 

.75 
.59 
.57 
.48 
1.03 

.62 
.37 
.61 
1.17 
.57 
1.09 

1.23 
.59 
.44 


Homogeneity 


1.7 


1.1 


5.3* 


13.5* 


13.8** 


3.8 
133.5 

113.7 

24.1 

30.0 

4.4 

104.9 

38.2 
99.5 

14.5 
119.1 

38.5 
34.9 
8.7 
17.2 
20.6 

40.3 
43.3 

4.5 
12.4 

4.8 

2.5 

0.2 
99.1 
25.9 


Note.  N  =  number  of  weighted  effect  size  (ES)  estimates  averaged.  If  95%  confidence 
interval  (CI)  does  not  contain  zero,  ES  is  significant  at  .05  level.  Qb  =  betwecn-group 
variance;  Qwi  =  group  variance  for  each  feature  sub-group.  "Significance  indicates 
rejection  of  hypothesis  of  between-group  homogeneity. 
*p<.05,  **E<.01 
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derived  from  studies  involving  multiple  centers  Qb(  1 )  =  5.3,  g  =  .02;  with  varied  response 
rates,  Q„(4)  =  18.8,  p  <  .001;  varied  country  sites,  Qb(4)  =  13.5,  e_=  .009;  and  year  of 
publication,  Qb(2)  =  13.8,  p  =  .001.  Studies  conducted  in  one  versus  two  or  more  centers 
represented  89%  of  Qb,  and  exhibited  significantly  greater  effect  size.  Effect  size 
magnitude  among  response  rates  stratified  in  decades  from  70%  was  similar  statistically. 
Highest  effect  size  was  reported  when  studies  did  not  report  response  rates.  The  results 
of  analyses  of  effect  size  variance  for  sample  design,  characteristics  of  control  group,  and 
whether  the  study  was  longitudinal  or  not,  revealed  no  conditional  variance  but  are 
presented  for  theoretical  or  empirical  reasons  previously  detailed. 

Substantive  features.  Substantive  features  are  summarized  in  Table  4-13. 
Despite  theoretical  predictions  discussed  in  Chapter  2,  no  significant  moderating  effects 
were  associated  with  the  taxonomy  of  PBL,  Qii{2)  =  1.8,  p  =  .19,  and  tutor  expertise, 
Qn(l)  =  2.2,  p  =  .33.  In  this  case,  statistical  insignificance  is  important,  particulariy  when 
an  instructional  characteristic  is  claimed  to  be  essential  by  advocates  of  problem-based 
learning  (Barrows,  1986,  2000).  Scope  of  problem-based  learning,  Qij(2)  =  5.0,  p  =  .03; 
institutions'  experience  with  problem-based  teaching  and  learning,  Qri(2)  =  9.7,  p  =  .02; 
students'  experience,  Qii(2)  =  16.2,  p  <  .001;  problem-type,  Qb(2)  =  10.3,  p  =  .02;  origin 
of  learning  issues,  Qb(2)  =  31.7,  p  <  .001;  and  degree  of  self-directed  learning,  Qb(2)  = 
1 8.0,  p  <.001  significantly  predict  effect  size.  However,  except  for  scope  of  PBL  and 
learning  issues  responsibility,  95%CI  of  effect  sizes  overlapped  for  sub-groups  within 
each  substantive  feature  of  problem-based  learning.  Effect  size  is  significantly  greater 
with  partial  course  (ES  =  1.33)  versus  curriculum-wide  (ES  =  .39)  or  entire  course  (ES  = 
.51 )  implementation  of  problem-based  instruction.  This  finding,  however,  is  derived 
from  one  report  (Itani,  Miller,  &  McCollum,  1997),  thus  the  implications  are  limited. 
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Table  4-13.  Mean  effect  size  estimates  and  variance  analyses  for  attitudes  toward 
learning  as  a  function  of  substantive  features 


N 

ES 

95%CI 

Homogeneity 

Variables 

Lower 

Upper 

Qa 

Qw, 

Scope  of  PBL 

5.0* 

Curriculum 

20 

.39 

.31 

.46 

77.1 

Entire  course 

9 

.51 

.36 

.65 

36.3 

Partial  course 

1 

1.33 

.95 

1.71 

0.0 

Institution's  PBL  experience 

9.7* 

0  years 

15 

.54 

.45 

.63 

96.5 

1-5  years 

5 

.32 

.12 

.53 

11.5 

6-10  years 

5 

.48 

.25 

.70 

7.0 

More  than  1 0  years 

5 

.34 

.14 

.54 

14.2 

Student's  PBL  experience 

16.2** 

0  years 

9 

.46 

.33 

.60 

9.6 

1  year 

8 

.64 

.50 

.77 

82.8 

2  or  more  years 

13 

.34 

.23 

.44 

30.3 

PBL  taxonomy 

1.8 

Reiterative  PBL 

22 

.43 

.36 

.50 

95.3 

Problem-based 

8 

.53 

.38 

.69 

38.1 

Problem  type 

10.3* 

Ill-structured 

12 

.52 

.41 

.63 

47.9 

Structured 

2 

.22 

-01 

.42 

0.1 

Mixed 

14 

.42 

.33 

.52 

80.4 

Learning  issues  responsibility 

31.7** 

Student  generated 

12 

.71 

.59 

.83 

39.4 

Faculty  generated 

2 

.22 

.01 

.42 

0.1 

Faculty-student  generated 

16 

.36 

.28 

.45 

67.8 

Tutor  expertise 

2.2 

Content  expert 

24 

.50 

.42 

.57 

118.1 

Peer 

2 

.31 

.04 

.57 

12.1 

Self-directed  learning 

18.0** 

Student  centered 

15 

.59 

.50 

.68 

51.7 

Plus  mandatory  lectures 

10 

.34 

.23 

.46 

34.7 

Plus  elective  lectures 

5 

.30 

.08 

.53 

34.6 

Note.  N  =  number  of  mean  effect  size  (ES)  estimates  averaged.  If  95 
interval  (CI)  does  not  contain  zero,  ES  is  significant  at  .05  level.  Qb  = 
variance;  Qw,  =  within-group  variance  for  each  feature  sub-group.  N 
independent  variables  because  some  features  could  not  be  ascertained, 
indicates  rejection  of  hypothesis  of  between-group  homogeneity. 
*p<.05.  **E<.01 


%  confidence 
:  between-group 
varies  among 
"Significance 


When  learning  issues  are  generated  exclusively  by  students  (ES  =  .71),  effect  size 
is  significantly  greater  than  when  generated  by  faculty  (ES  =  .22)  or  a  hybrid  of  faculty 
and  students  (ES  =  .36). 


203 

Table  4-14  represents  the  correlation  matrix  of  hypothesized  positive  predictive 

dimensions  for  PBL  efficacy  as  discussed  in  Chapter  2.  Learning  issues  responsibility  is 

the  only  problem-based  leaning  dimension  coiTelated  with  attitude  toward  learning  effect 

size  (r  =  .39,  p  =  .04).  Multicolinearity  is  evident  among  dimensions  including  between 

taxonomy  of  PBL  and  tutor  expertise  (r  =  .48,  p  =  .01)  and  between  learning  issues 

responsibility  and  self-directed  learning  (r  =  .41,  p=  .03). 

Table  4-14.  Intercorrelations  in  PBL  dimensions  for  attitude  toward  learning 
Variable  A  B  C  D  E 


Mean  effect  size  estimate 

-.09 

.05 

.39* 

.06 

.28 

A.  PBL  taxonomy 

.07 

.34 

.48* 

.30 

B.  Problem  type 

.11 

-.33 

.22 

C.  Learning  issues  responsibility 

.25 

.41* 

D.  Tutor  expertise 

.29 

E.  Self-directed  learning 

Note.  Data  are  derived  from  30  studies.  PBL  taxonomy  =  reiterative  or  not;  Problem 
type  =  ill-structured  or  not;  learning  issues  responsibility  =  student  derived  or  not;  Tutor 
expertise  =  content  expert  or  not;  Self-directed  learning  =  student  centered  or  not. 
*2<.10. 

Examination  of  Outlier  Study 

Blumberg  and  Daugherty  (1994)  compared  student  attitudes  toward  learning  after 
one  and  two-years  of  preclinical  education  in  parallel  paths  of  lecture-based  versus 
problem-based  curriculum  for  two  consecutive  classes  at  Rush  Medical  College. 
Substantive  characteristics  of  lecture-based  and  problem-based  curricula  at  Rush  Medical 
College  previously  were  reported  with  biomedical  knowledge  achievement.  A  Likert- 
scaled  instrument  (Educational  Activities  Utility  Survey)  was  designed  to  examine 
student  attitudes  regarding  the  effectiveness  of  their  medical  education  in  preparing  them 
for  passing  examinations  and  becoming  competent  physicians.  Students  from  previous 
classes  piloted  the  instrument  to  reflect  variance  estimates  for  each  curriculum.  First  year 
students'  response  rate  was  64%  (1 15/180)  and  84%  (32/38)  from  the  lecture-based  and 
problem-based  curricula,  respectively.  Second  year  students'  response  rate  was  60% 
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(108/180)  and  65%  (21/35)  from  the  lecture-based  and  problem-based  curricula, 
respectively.  Because  responses  to  survey  items  were  "highly  correlated"  (p.  88) 
between  the  two  first  year  classes  in  each  curriculum,  responses  were  pooled  for 
analyses.  Reported  correlations  range  from  .78  to  .94.  Academic  and  demographic 
characteristics  of  respondents  and  non-respondents  were  compared  and  found  to  be 
statistically  equivalent.  Statistical  evidence  for  intergroup  baseline  equivalence  is 
provided.  For  LBL  students,  the  correlation  between  their  perceived  effectiveness  of 
cunicular  activities  for  passing  both  their  examinations  and  becoming  a  competent 
physician  was  r  =  .07.  For,  PBL  students,  the  correlation  between  perceived 
effectiveness  of  their  education  for  passing  both  their  assessments  and  becoming  a 
competent  physician  was  r  =  .83.  Interestingly,  students  undergoing  a  lecture-based 
curriculum  perceived  no  relationship  between  their  educational  activities  and  either 
passing  examinations  or  becoming  a  competent  physician.  Correlation  coefficients  were 
converted  to  an  effect  size  estimate  of  1 .76  (95%CI  1 .25  to  2.27)  using  the  methods 
described  in  Chapter  3. 
Examination  of  Select  Included  Studies 

Bernstein  et  al.  (1995)  investigated  changes  in  second  year  medical  students' 
attitudes  from  initial  and  end  of  course  experience  with  problem-based  learning.  Neither 
the  students  nor  their  institution  had  any  experience  with  problem-based  teaching  and 
learning.  Indeed,  an  aim  of  the  study  was  to  develop  guidelines  to  implement  PBL  at  the 
University  of  Toronto.  Attitudes  toward  learning  were  quantified  using  Likert  scaled 
items  with  dimensions  including  knowledge,  clinical  skills,  teamwork,  and  relating  to 
patients.  Qualitative  responses  to  open-ended  questions  were  coded  to  discover  emerging 
themes.  Seventy-eight  percent  (196/250)  and  83%  (207/250)  completed  pre-test  and 
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post-test  questionnaires.  Overall,  there  was  a  significant  change  in  students'  perception 
from  38%  (pre-test)  to  52%  (post-test)  agreement  that  problem-based  learning  is  more 
effective  than  lecture-based  learning  (chi-squared  =  6.5,  e  <  .02).  The  differential  effects 
of  PBL  regarding  the  specific  dimensions  of  attitudes  toward  learning  included:  (a) 
students  rated  lecture-based  learning  as  better  for  knowledge  acquisition,  and  (b)  students 
rated  PBL  better  for  improving  teamwork  and  relationships  with  patients.  Themes 
emerging  from  qualitative  analyses  included:  (a)  concents  that  PBL  would  result  in 
knowledge  gaps,  reinforce  misconceptions,  and  was  an  inefficient  use  of  time;  (b) 
students  suggested  a  combination  of  problem-based  and  lecture-based  methods,  or  a 
gradual  transition  to  PBL;  (c)  the  PBL  process  raises  issues  not  normally  discussed;  and 
(d)  because  they  were  expected  to  participate,  students  acknowledged  discussion  forced 
them  to  think  about  concepts  more  thoroughly  and  facilitated  greater  understanding  and 
retention.  This  latter  observation  is  consistent  with  some  perspectives  of  social 
constructivism.  For  example.  Brown  and  Palinscar  (1989)  argue  learning  is  enhanced 
when  students  are  required  to  explain,  elaborate  or  defend  their  stance  to  others. 
Constructing  an  explanation  (dialectical  constructivism,  Moshman,  1982)  might  induce 
students  to  integrate  and  elaborate  knowledge  in  novel  ways.  Moreover,  Dewey 
(1933/1998)  advocated  recitation  as  a  path  to  individual  and  shared  understanding.  This 
study  suggests  intellectual  responsibility,  sine  qua  non  for  efficacious  collaborative 
learning,  is  fostered  by  problem-based  instruction. 

Kaufman  and  Mann  (1996)  compared  second-year  medical  sttidents'  attitudes 
toward  their  learning  environment  in  the  inaugural  PBL  class  with  those  in  the  final  LBL 
class  at  Dalhousie  University.  The  1995  graduating  class  underwent  traditional,  lecture- 
based  preclinical  education.  The  1996  graduating  class  inaugurated  the  problem-based 
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learning  curriculum.  Weekly  lecture  time  was  reduced  from  about  twenty  to  three  to  five 
hours.  Students  participated  six  hours  weekly  in  faculty-tutored  small-groups 
approximating  Barrows'  reiterative  problem-based  teaching  and  learning  model  (see 
Chapter  2).  Attitude  toward  learning  was  quantified  with  a  12-item  instrument  with 
Likert  scaled  responses  ranging  from  1  (small  extent)  to  9  (large  extent).  Students  were 
asked  "to  rate  the  extent  to  which  your  courses  were  characterized  by  the  following 
features":  learning  details;  understanding  principles  and  being  able  to  use  them; 
integrating  different  subjects  in  order  to  solve  problems;  articulating  previous  knowledge; 
stating  learning  objectives;  making  decisions;  independent  thinking;  problem  solving; 
gathering  and  analyzing  information;  stimulafing  and  enjoyable;  stimulated  to  learn  more; 
and  stimulated  to  read  medical  literature.  Response  rates  were  85%  (73/86)  and  87% 
(72/83)  for  LBL  and  PEL  students,  respectively.  Ratings  were  significantly  greater  (p  < 
.001)  by  PEL  students  for  1 1  of  12  items.  There  was  no  differential  effect  of 
instructional  format  regarding  the  "making  decisions"  feature.  Standardized  and 
weighted  effect  size  was  computed  for  each  item  using  presented  group  size,  M  and  SD 
and  pooled  for  summary  (ES  =  1 . 1 8,  95%CI  0.58  to  1 .26).  Data  for  the  "learning  details" 
item  were  reversed  for  effect  size  calculation  to  reflect  a  positive  finding  for  lower  score 
(Lipsey  &  Wilson,  2001).  Inferences  from  this  study  are  affected  by  threats  to  internal 
validity  (history  and  tnaturation)  and  external  validity  (selection  bias).  Although  the 
authors  statistically  substantiate  baseline  equivalence  of  groups  with  academic  and 
demographic  data,  nonparallel  cohorts  are  susceptible  to  historical  and  maturation 
changes  that  might  affect  attitude  toward  learning.  Moreover,  because  students  were 
studied  in  sequential  curricular  cohorts,  the  results  may  not  be  representative. 
Notwithstanding  these  limitations,  the  significance  of  effect  size  is  impressive  and 
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consistent  with  predictions  advanced  by  some  social  constructivists  (Brown  &  Palinscar, 
1989;  Cobb,  2002;  Mosliman,  1982;  Savery  &  Duffy,  1995)  and  advocates  of  situated 
learning  (Brown  et  al,  1989;Greeno,  1998;  Lave  &  Wenger,  1991). 

Extending  their  work,  Mann  and  Kaufman  (1999)  studied  graduates  of  lecture- 
based  (1995)  and  problem-based  learning  (1996)  curricula  al  Dalhousie  University  in 
their  second  year  of  residency  training.  This  study  involved  the  same  cohorts  as  their 
previous  work,  but  approximately  four  years  later  during  residency  training  (Kaufman  & 
Mann,  1996).  A  22-item,  Likert-scaled  (1  =  unsatisfactory  to  7  =  excellent)  instrument 
incorporated  sub-scales  designed  to  quantify  physicians'  attitudes  toward  the  utility  of 
their  education  to  prepare  them  for  practice  competence.  Addressed  competencies 
included  integration  of  biomedical  and  clinical  science;  adequacy  of  knowledge  base  and 
ability  to  apply  knowledge;  clinical  reasoning  skills;  self-directed  learning  skills; 
obtaining,  analyzing,  and  synthesizing  information  to  resolve  clinical  problems; 
assessment  of  novel  situations;  use  of  information  technology;  identification  and 
evaluation  of  health  problems;  and  teamwork.  The  instrument  was  piloted  with  several 
faculty  to  assess  congruency  with  educational  goals,  then  with  residents,  who  had 
graduated  from  other  medical  schools,  to  assess  clarity  and  coherence.  The  response  rate 
was  66%  (51/84)  and  77%  (49/64)  for  graduates  of  lecture-based  and  problem-based 
learning  curriculum,  respectively.  Data  were  analyzed  using  a  two-factor  (Factor  1  = 
curriculum;  Factor  2  =  residency  specialty)  analysis  of  variance.  Sub-scale  reliability 
was  assessed  with  Cronbach's  alpha.  Construct  validity  was  assessed  by  comparing 
attitude  sub-scales  with  performance  on  the  Medical  Council  of  Canada  Medical 
Qualifying  Examinations.  Alpha  coefficients  ranged  from  .83  to  .85.  Correlations 
assessing  construct  validity  ranged  from  .71  to  .98.  There  were  no  significant 
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interactions.  There  were  no  significant  main  effects  (g  =  .23),  nor  intergroup  differences 
in  any  attitude  toward  learning  sub-scale.  Thus  summary  data  are  pooled  for  PBL  (M  = 
4.91 ,  SD  =  1 .27)  and  LBL  (M  =  5.22,  SD  =  1 .23)  producing  a  standardized  effect  size 
estimate  of -0.04  with  a  95%C1  -0.44  to  0.40.  Interpretation  of  the  results  may  be 
confounded  by  selection  bias,  although  the  authors  claim  baseline  equivalence  due  to 
statistically  similar  academic  and  demographic  characteristics.  The  authors  suggest  PBL 
students  had  graduated  from  a  new  program,  thus  they  may  not  be  representative. 
Findings  of  the  present  research  do  not  support  that  argument,  as  there  appears  to  be  no 
consistent  relationship  between  institutional  experience  and  problem-based  learning. 

Lieberman  et  al.  (1997)  compared  the  differential  effects  in  attitude  toward 
learning  in  students  undergoing  parallel  PBL  and  LBL  curricula  at  the  University  of 
Texas  Medical  Branch-Galveston.  Students  undergoing  LBL  experienced  preclinical 
education  via  twenty-four  to  thirty  hours  of  lectures  weekly.  Students  in  the  problem- 
based  curriculum  path  experience  preclinical  education  in  tutorial  groups  (n  =  6)  meeting 
two-hours  three  times  weekly  facilitated  by  biomedical  science  and  clinical  faculty  tutors. 
Although  core  content  was  similar,  knowledge  for  LBL  students  was  primarily  assessed 
with  recognition  examinations  (e.g.,  multiple-choice  questions),  whereas  PBL  underwent 
multi-method  assessments  (e.g.,  tutor  assessments,  standardized  patient  examinations, 
short  essay  question  examinations).  Student  attitudes  toward  three  dimensions  of  the 
learning  environment  including  teaching  and  administration,  psychological  and  emotional 
aspects  of  student  life,  and  intellectual  atmosphere  were  assessed  with  MSLES  and  CBS. 
Students  were  asked  to  complete  equivalent  forms  of  surveys  at  the  beginning  and  end  of 
their  first  preclinical  year.   Initial  and  final  response  rates  from  LBL  and  PBL  students 
were  97%  (176/180)  and  100%  (24/24)  and  59%  (106/180)  and  100%,  respectively. 
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Student  groups  were  statistically  equivalent  regarding  acadeinic  and  sociodemographic 
characteristics  at  baseline  and  follow-up.  Cronbach's  alpha  coefficients  for  MSLES 
ranged  from  .55  to  .76.  Only  the  positive  learning  experience  sub-scale  of  CBS  was 
assessed  and  it  exhibited  a  reliability  coefficient  of  .86.  Measures  of  first  year  change  in 
positive  attitudes  for  LBL  students  revealed  a  significant  decline  in  every  sub-scale,  with 
the  lowest  level  of  significance  at  <  .05.  Investigators  observed  significant  first  year 
increases  in  six  of  eight  attitude  sub-scales  in  PEL  students,  with  the  least  level  of 
significance  at  <  .01 .  Notable  curricular  affects  aligned  with  the  aims  of  problem-based 
teaching  and  learning  included  both  sub-scales  of  the  attitude  toward  intellectual  climate. 
Meaningful  learning  experience  (M  =  -.39,  SD  =  .05,  p  <  .001,  scale  range  1-5)  and 
positive  learning  experience  (M  =  -3.94,  SD  =  .69,  p  <  .001,  scale  range  9-63)  declined  in 
students  undergoing  the  leeture-based  curriculum.  Meaningful  learning  experience  (M  = 
.53,  SD  =  .09,  g<  .001)  increased  and  positive  learning  experience  (M  =  2.92,  SD  =  2.12) 
was  unchanged  in  students  undergoing  the  problem-based  curriculum.  1  computed  effect 
size  estimates  for  each  subscale,  and  then  they  were  averaged  for  the  study.  Effect  size  of 
0.97  is  significant  because  the  associated  95%  confidence  interval  0.51  to  1.43  does  not 
contain  zero. 

In  a  follow-up  longitudinal  study  with  similar  methodology,  Lieberman  and 
colleagues  (2001)  sought  to  discover  whether  positive  attitudes  induced  by  the  problem- 
based  curriculum  were  sustained  beyond  the  first  year  of  medical  school.   Matriculating 
students  in  the  classes  of  1995  and  1996  undergoing  LBL  and  PBL  completed  the 
MSLES  and  CBS  at  the  end  of  the  first  year  and  end  of  years  two  and  four  of  medical 
school.  Complete  data  were  available  for  73%  (259/357)  and  87%  (41/47)  of  LBL  and 
PBL  students,  respectively.  Data  were  compared  with  a  two-factor  analysis  of  variance 
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(curriculum  and  lime)  and  multiple  regression  analyses.  Intergroup  differences  in 
academic  and  sociodemographic  characteristics  were  equivalent  statistically.  No 
significant  differences  were  revealed  between  students  who  completed  and  did  not 
complete  the  study.  Positive  learning  experience  measures  (scale  9  -  63)  were 
statistically  similar  in  PBL  versus  LBL  students  at  baseline  (M  =  50.0,  SD  =  7.3  vs.  M  = 
47.6,  SD  =  7.9),  year-two  (M  =  50.0,  SD  =  5.6  vs.  M  =  44. 1,  SD  =  8.6),  and  year-four  (M 
=  50.9,  SD  =  5.5  vs.  M  =  47.6,  SD  =  7.8).  Within  the  cuniculum  analyses  revealed  in 
LBL  students,  a  significant  decline  in  attitude  toward  positive  learning  experience 
between  baseline  and  second-year  (p  <  .05),  but  a  return  to  baseline  level  at  year-three. 
However,  baseline  attitude  toward  positive  learning  environment  was  sustained  in  PBL 
students  throughout  the  study.  Meaningful  learning  experience  measures  (scale  1-5)  were 
statistically  different  in  PBL  versus  LBL  students  at  baseline  (M  =  3.8,  SD  =  0.5  vs.  M  = 
3.5,  SD  =  0.5,  2  <  .05),  year-two  (M  =  4.0,  SD  =  0.5  vs.  M  =  3.1,  SD  =  0.5,  p  <  .05),  and 
year-four  (M  =  3.5,  SD  =  0.4  vs.  M  =  3.3,  SD  =  0.5,  p  <  .05).  Within-curriculum 
analyses  in  LBL,  students  revealed  a  significant  decline  in  attitude  toward  positive 
learning  experience  between  baseline  and  second-year  (2  <  .05)  and  baseline  and  fourth- 
year  (g  <  .05).  In  PBL,  there  was  no  observed  change  between  baseline  and  second-year, 
but  there  was  a  significant  decline  between  baseline  and  fourth-year  (g  <  .05)  attitude 
toward  positive  learning  experience.  Problem-based  instruction  induced  change  in 
measured  attitudes  toward  learning  but  was  sustained  only  for  meaningful  learning 
experiences.  Problem-based  teaching  and  learning  induced  attitude  for  positive  learning 
experience  was  not  sustained  during  clinical  science  training  (i.e.,  clerkships).  Clearly, 
educational  goals  vis-a-vis  attitudes  toward  learning  are  realized  better  in  students 
undergoing  reiterative  PBL  versus  LBL  curriculums,  but  after  two-years  of  common 
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clerkships,  some  differential  gains  vanish.  Extending  into  clerkship  years  the  elements  of 
problem-based  teaching  and  learning  responsible  for  fostering  positive  attitudes  toward 
learning  would  appear  to  be  warranted. 


CHAPTER  5 
SUMMARY  AND  CONCLUSIONS 

The  research  presented  here  used  meta-analytic  methods  to  compare  cognitive  and 
non-cognitive  outcomes  in  medical  students  and  physicians  who  underwent  problem- 
based  learning  versus  lecture-based  leaming.  Meta-analysis  also  provided  an  opportunity 
to  quantify  variance  of  methodological  and  substantive  features  across  studies  to  identify 
potential  mediators  of  observed  instructional  effects.  Research  design  and  interpretation 
of  results  were  infomied  by  a  theoretical  framework  incorporating  educational 
philosophy,  information-processing  theory,  and  principles  of  social  constructivism  and 
situated  leaming.  Eighty-two  studies  with  121  effect  size  estimates  reported  between 
November  1977  and  December  2002,  involving  forty-five  tmiversities  in  twelve  countries 
and  12,979  participants  (PBL  =  4,227;  LBL  =  8,752)  were  included  in  this  research. 
Findings  and  implications  of  the  main  effects  and  variance  analyses  for  biomedical 
science  achievement  (n  =  33),  clinical  science  achievement  (n  =  29),  problem  solving 
achievement  (n  =  10),  self-directed  learning  skills  (n  =  19),  and  attitude  toward  leaming 
(n  =  30)  are  presented  relative  to  specific  research  questions.  Conclusions  and 
recommendations  for  future  research  are  presented  as  well. 

Questions  Answered  by  this  Research 
Research  question  1.  Is  there  a  significant  positive  effect  size  at  the  .05  level  for 
biomedical  science  achievement  in  students  and  physicians  who  undergo  medical 
education  with  PBL  compared  to  LBL?  There  was  no  significant  difference  in  the 
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weighted  mean  effect  size  estimate  for  biomedical  science  achievement  (ES  =  .02,  95%CI 
-.02  to  .06)  in  33  independent  research  samples  involving  12,298  participants.  Sensitivity 
analysis  revealed  that  the  one  outlier  had  little  impact  on  mean  effect  size  and  there  was 
no  evidence  of  publication  bias  (Spearman's  rho  =  .12,  g  =  .51).  However,  findings 
should  be  interpreted  with  caution  as  probability  of  a  Type  II  error  is  73  percent. 

Given  its  theoretical  underpinnings,  failure  to  reveal  better  efficacy  for  problem- 
based  instruction  is  curious.  On  the  other  hand,  discovering  a  statistically  equivalent 
instructional  efficacy  for  biomedical  science  achievement  is  an  important  practical 
finding.  As  discussed  in  Chapter  1,  traditionalists  and  opponents  of  problem-based 
teaching  and  learning  argue  that  biomedical  science  facts  and  concepts  are  conveyed  best 
via  discipline-based  lectures.  In  particular,  some  educators  contend  that  by  focusing  on 
the  learning  process  (mind)  rather  than  the  content  (memory)  in  a  student-directed 
learning  environment,  problem-based  instruction  results  in  a  curriculum  they  label  as 
"basic  science  lite"  (Swanson  &  Case,  1997,  p.  73).  The  present  research  suggests  that 
problem-based  instruction  is  at  least  as  efficacious  as  lecture-based,  discipline-oriented, 
instruction  for  effecting  the  acquisition  of  biomedical  science  knowledge  as  indicated  by 
results  of  licensing  examination.  The  educational  significance  of  this  finding  is  important 
because  variance  analyses  of  substantive  features  suggest  that  problem-based  instruction 
operationally  values  collaborative  learning  and  student  empowerment  (e.g.,  student- 
generated  learning  issues)  while  fostering  self-directed  learning.   From  a  philosophical 
perspective,  given  the  equivocal  evidence  and  confronted  with  an  either/or  decision, 
theoretical  grounding  might  be  the  tiebreaker  for  choice  of  instructional  approach,  thus 
favoring  problem-based  teaching  and  learning.  Support  for  this  stance  is  found  in  Dewey 
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(1929b)  who  wrote:  "Theory  is  in  the  end,  as  has  been  well  said,  the  most  practical  of  all 
things"  (p.  17). 

There  are  at  least  three  potential  reasons  why  biomedical  science  achievement  is 
no  more  superior  with  problem-based  instruction  than  with  lecture-based  instruction. 
These  include:  (a)  instructional  efficacy  converges  with  respect  to  biomedical  science 
achievement;  (b)  problem-based  instruction  may  not  be  engineered  sufficiently  to  alter 
the  learning  process  and  outcomes  because  learners  fail  to  develop  appropriate  learning 
strategies;  or  (c)  assessments  of  achievement  may  be  insensidve  to  the  nature  of  changes 
in  learning.  These  issues  are  addressed  in  the  following  section. 

Research  question  2.  Is  variation  in  aggregated  effect  size  estimates  for 
biomedical  science  achievement  greater  than  the  sampling  error  warranting  sub-group 
analyses  to  test  for  potential  moderation  by  a  priori  defined  outcome  measures, 
methodological  features,  and  substantive  features  of  problem-based  teaching  and 
learning?  Variance  analyses  revealed  significant  heterogeneity  in  biomedical  science 
achievement  effect  size  estimates  (Q  =  133.6,  g  <  .00001).  Examination  of  theoretically 
and  empirically  grounded  potential  moderators  of  effect  size  estimates  suggests  that 
variation  in  outcome  measures,  research  methodologies,  and  substantive  features  of 
problem-based  instruction  across  studies  infiuences  effect  size  estimates.  These  findings 
provide  a  substrate  to  suggest  the  etiology  of  confiicting  reports.  For  example,  measuring 
biomedical  science  achievement  using  licensing  examinations  or  locally  constructed 
assessments  produces  significantly  different  instructional  effects.   If  differential 
mstructional  effects  are  assessed  using  standardized  examinations,  students  undergoing 
lecture-based  instruction  tend  to  be  favored.  However,  when  cohorts  are  compared  using 
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data  from  non-standardized,  criterion-referenced  assessments  in  general  and  multi- 
method  item  assessments  specifically,  students  undergoing  problem-based  instruction 
tend  to  exhibit  the  greatest  gain  in  biomedical  science  knowledge.  In  the  only  previous 
meta-analysis,  Venion  &  Blake  (1993)  found  a  weighted  mean  effect  size  of -0.18 
(95%C1  -0.26  to  -0.10)  and  -0.09  (95%CI  -0.24  to  0.06)  for  licensing  examinations  (N  = 
5,  all  were  NBME  Part  1)  and  other  measures  (N  =  5,  mostly  MCQ  examinations)  of 
biomedical  science  achievement,  respectively.  The  present  research  extracted  data  from 
seventeen  studies,  including  the  five  studies  examined  by  Vernon  and  Blake,  and 
discovered  67%  less  negative,  and  statistically  equivalent  instructional  effect  when 
compared  with  licensing  examination  scores  (ES  =  -0.06,  95%CI  -0.13  to  0.01). 
However,  significant  variance  heterogeneity  of  aggregated  effect  sizes  warranted 
subgrouping  by  outcome  measures.  Further  analyses  of  instructional  contrasts  with 
sixteen  non-licensing  assessment  data  sets  (ES  =  0.07,  95%CI  0.02  to  0.13)  revealed  a 
small  positive,  but  statistically  significant,  instructional  effect  favoring  problem-based 
over  lecture-based  teaching  and  learning.  Given  probability  of  Type  11  error  associated 
with  main  effect,  this  finding  statistically  warrants  inference  that  problem-based  versus 
lecture-based  learning  affects  equal  or  greater  biomedical  science  achievement. 
Implications  are  that  these  findings  should  allay  concerns  medical  educators  have 
regarding  the  licensing  examination  performance  of  students  undergoing  problem-based 
teaching  and  learning. 

In  1992,  NBME  Part  I  was  replaced  with  USMLE  Step  1.  The  USMLE  Step  1  is 
designed  to  assess  similar  biomedical  science  knowledge  with  a  similar  number  of 
multiple-choice  questions,  but  incorporates  a  greater  percentage  of  questions  deigned  to 
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tap  higher-order  learning  rather  than  factual  knowledge  exclusively  (O'Donnell  et  al., 
1993).  Theoretically,  PBL  students  are  advantaged  by  the  shift  in  objectives  and  item 
construction  of  licensing  examinations.  However,  despite  increased  emphasis  on  the 
evaluation  and  application  of  biomedical  science  knowledge,  the  sensitivity  of  USMLE 
Step  1  to  detect  relative  instructional  effects  has  been  questioned  (Ripkey  et  al.,  1998). 
Conducting  a  random  effects  analysis  of  variance,  Ripkey  et  al.  observed  from  1994  to 
1998,  89%  of  total  variation  in  USMLE  Step  1  scores  for  54,890  participants  from  118 
medical  schools  was  within  schools  rather  than  between  schools,  while  1 1%  was  between 
schools.  Thus,  very  little  of  the  variation  in  a  school's  USMLE  Step  1  scores  was 
explained  by  instructional  format.  Unfortunately,  the  sample  size  for  problem-based 
curricula  was  too  small  for  independent  analyses.  Indeed,  only  three  investigations 
contrasting  achievement  gain  with  USMLE  Step  1  were  available  for  the  present  study. 
Although  two  studies  from  Wake  Forest  University  School  of  Medicine  compared  the 
curricular  effects  of  LBL  and  PBL  on  biomedical  knowledge  achievement  as  assessed 
with  NBME  Part  I  scores  (Richards  &  Cariago,  1993)  and  USMLE  Step  1  scores 
(Enarson  &  Carrigo-Lo,  2001 ).  Richards  and  Cariago  observed  no  difference  in  NBME 
Part  I  scores  between  LBL  and  PBL  students  completing  the  examination  in  1991. 
Similarly,  Enarson  and  Carrigo-Lo  discovered  no  difference  in  USMLE  Step  1  scores 
between  LBL  and  PBL  students  completing  the  examination  each  year  from  1992  to 
1998.  Since  current  findings  suggest  instructional  contrasts  with  locally  constructed 
assessments  (criterion-referenced)  evidence  a  significant  positive  effect  size,  the 
construct  validity  of  the  USMLE  Step  I  could  be  challenged.  Considering  the  potential 
dominant  effect  of  licensing  examinations  on  medical  student  learning,  further  research  is 
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warranted  to  examine  whether  conversion  to  USMLE  Step  1  from  NBME  Part  I  has 
accomplished  the  desired  assessment  goals. 

Since  problem-based  instruction  is  designed  to  influence  the  quality  (expertise) 
rather  than  the  quantity  (encyclopedic)  of  learning,  data  derived  using  assessments  not 
designed  to  capture  the  differential  quality  of  conceptual  knowledge  are  antithetical  to  the 
aims  of  problem-based  learning  and  contemporary  cognitive  science  and  educational 
psychology.  Consistent  with  perspectives  emerging  from  both  situational  learning 
(Brown  et  al,  1989;  Brown  &  Campione,  1990;  Brown  &  Palinscar,  1989;  Greeno,  1998; 
Lave  &  Wenger,  1991)  and  information-processing  theory  (Anderson,  1990,  1993)  as 
discussed  in  Chapter  2,  problem-based  teaching  and  learning  assumes  that  medical 
knowledge  is  multidimensional  by  nature,  thus  employing  a  standardized  or  one- 
dimensional  assessment  appears  to  be  inadequate  (Barrows,  2000;  Shepard,  2000).  The 
present  investigation's  findings  confirm  that  assessing  a  multidimensional  construct 
requires  multimethod  assessments  to  capture  effectively  the  desired  learning  outcomes. 
Multimodal  assessments  target  learning  processes,  learning  products,  and  their  interaction 
(Bloom,  1984;  Boud,  1992;  Shepard).  In  particular,  valid  assessments  should  assay  the 
extent  to  which  a  student's  present  understanding  provides  the  scaffolding  necessary  for 
the  application,  synthesis,  and  evaluation  of  novel  problems  in  the  same  domain  (Bloom). 
Such  a  teaching  and  learning  framework  converges  nicely  with  recent  enthusiasm  for 
interactive  assessment.  As  discussed  in  Chapter  2,  interactive  assessment  permits  tutors 
to  provide  assistance  as  part  of  assessment  (Shepard).  For  Shepard,  assessment  is 
embedded  in  the  process  of  teaching  and  learning.  Within  this  framework,  interactive 
assessment  creates  targeted  opportunities  for  tutors  to  identify  misconceptions  and  gain 
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insight  about  facilitating  understanding.  Moreover,  interactive  assessment  may  facilitate 
opening  the  "black  box"  of  learning  (Dubin  &  Taveggia,  1968)  to  discover  the  structure 
and  function  of  cognitive  representations  (Anderson).  Ultimately,  as  a  research  method, 
interactive  assessments  might  be  used  to  clarify  the  process  of  concept  formation 
(Gal'perin,  1969)  and  conditions  of  cognition  (Glasser,  1991 ).  Furthermore,  such  a 
method  might  reveal  the  critical  threshold  of  biomedical  science  knowledge  required  to 
provide  sufficient  competence  to  proceduralize  knowledge  (Anderson)  and  facilitate 
meta-cognition  and  self-directed  learning  (Winne,  1995a,  1995b;  Winne  &  Hadwin, 
1998).  If  the  goal  of  medical  education  research  is  to  measure  the  process  of  learning 
instructional  objectives,  further  research  is  required  to  develop  more  instructionally 
sensitive  assessments  (Crocker  &  Algina,  1986)  to  account  for  what  has  been  learned  in 
both  experimental  and  control  medical  instructional  environments. 

Another  explanation  for  not  revealing  a  difference  in  overall  effect  in  biomedical 
science  achievement  is  there  actually  may  not  be  a  differential  instructional  effect. 
Possibly  both  lecture-based  and  problem-based  instructional  methods  converge 
efficaciously  with  respect  to  facilitating  biomedical  science  knowledge  acquisition.  This 
is  an  unlikely  explanation  given  the  observed  differential  effects  of  norm-referenced  and 
criterion-referenced  assessments.  Clearly  instructional  format  affects  biomedical  medical 
science  achievement  when  assessments  are  congruent  with  learning  objectives. 

Another  important  finding  is  that  the  magnitude  of  effect  size  estimates  did  not 
differ  as  a  function  of  studies  with  varied  scopes  of  problem-based  instruction.  This 
observation  is  valuable  because  some  researchers  argue  that  interventions  not 
implemented  curriculum  wide  are  unlikely  to  be  efficacious  (Colliver,  2000).  For 
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example,  Colliver  argues,  "for  PBL  to  be  effective  the  entire  curriculum  must  be  problem 
centered  and  PBL  should  form  the  core  of  the  curriculum"  (p.  260).  Consequently,  in  his 
literature  review,  Colliver  did  not  include  comparative  research  unless  both  problem- 
based  instruction  and  lecture-based  instruction  were  implemented  curriculum-wide. 
Current  findings  suggest  discarding  studies  not  involving  curriculum-wide  instruction 
would  exclude  important  data  that  does  not  bias  results.  Moreover,  analysis  of  studies 
conducted  in  hybrid  instructional  environments  suggests  that  the  aims  of  problem-based 
instruction  einerge.  Indeed,  such  a  research  setting  may  facilitate  unpacking  the  best 
operational  features  of  each  instructional  approach.  This  will  be  detailed  later. 

A  counterintuitive  discovery  was  the  lack  of  apparent  association  between 
institutional  experience  with  problem-based  instruction  and  improved  biomedical  science 
achievement.  This  finding  should  be  encouraging  for  curriculum  designers,  because 
problem-based  instruction  appears  to  be  an  innovation  capable  of  affecting  positive 
cognitive  change  relatively  quickly.  In  particular,  acute  conversion  to  problem-based 
instruction  does  not  lead  to  a  reduced  ability  to  pass  national  licensing  examinafions 
assessing  biomedical  science  knowledge. 

The  Hawthorne  effect  is  a  legitimate  competitive  explanation  for  this  observafion. 
Unfortunately,  only  a  few  reports  permitted  assessment  of  validity  threat  via  Hawthorne 
effect.  For  example,  Blake  and  associates  (2000)  observed  after  curriculum-wide 
implementation  of  PBL,  students  exceeded  both  an  historical  control  and  North  American 
mean  USMLE  Step  I  scores,  in  three  out  of  four  years.  In  this  instance,  sustained  or 
unproved  achievement  for  three  consecutive  years  makes  conventional  interpretation  of  a 
Hawthorne  effect  an  unlikely  explanation.   Even  if  these  effects  could  be  attributed 


220 

primarily  to  greater  expectations  and  efforts  accompanying  novelty  of  problem-based 
instruction,  these  results  suggest  cognitive  advantages  can  be  realized  relatively  rapidly 
and  persist  following  conversion  to  problem-based  teaching  and  learning.  That  is,  an 
outcome  attributed  to  a  sustained  Hawthonie  type  effect,  similar  to  a  persistent  placebo 
effect  in  pharmaceutical  research,  might  be  viewed  as  a  desirable  and  valid  outcome. 
Indeed,  because  vast  majority  of  medical  students  have  been  enculturated  in  traditional, 
syllabus  driven  pedagogical  methods  throughout  their  primary,  secondary  and  collegiate 
education,  I  suspect  Hawthorne  effect  operates  as  a  bona  fide  dimension  of  problem- 
based  teaching  and  learning.  Detailed  examination  of  longitudinal  and  cross-sectional 
research  should  differentiate  conventionally  operationalized  Hawthorne  effects. 
Although  few  in  number,  findings  of  this  research  suggest  no  significant  differential 
effect  when  instructional  contrasts  were  cross-sectional  or  longitudinal. 

Another  plausible  explanation  is  that  implementation  of  problem-based 
instruction  may  not  be  sufficient  to  alter  learning  process  and  outcomes  because  learners 
fail  to  develop  appropriate  learning  strategies  (Shepard,  2000;  Shulman,  1986). 
hicumbent  upon  educators  is  to  know  whether,  and  how,  students  adapt  their  learning 
strategies  in  response  to  changes  in  instructional  design.  Few  primary  research  reports 
provide  confirmation  that  intended  learning  processes  are  realized.  Cleariy,  learning 
process  effected  by  instruction  must  be  confinned  before  outcome  variables  can  be 
appropriately  interpreted  and  generalized.   If  problem-based  instruction,  or  any 
pedagogical  approach,  is  to  enhance  learning,  it  must  change  more  than  means  learners 
acquire  information  (Shepard).  To  interpret  adequately  interaction  between  instruction 
and  outcomes,  researchers  must  examine  how  learners  process  information  and  construct 
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knowledge  (Shepard).  As  argued  in  Chapter  2,  interactive  assessment  methods  may 
facilitate  understanding  between  teaching  and  learning,  thus  providing  a  method  for 
evidenced-based  teaching  and  learning  (Shepard).  This  would  be  particularly  evident  if 
such  assessment  incorporated  an  information  processing  framework  (Anderson,  1990, 
1993a)  discovered  by  Pate!  and  associates  (Patel  et  al.,  1988;  Pate),  Groen,  & 
Frederiksen,  1986)  to  provide  a  method  of  analyzing  knowledge  organization, 
understanding  and  retrieval  in  medical  students  and  physicians  experimentally  challenged 
with  clinical  problems.  This  line  of  research  in  medical  education  could  reveal  materia 
medica  of  pedagogy  (Shulman)  and  inform  the  art  and  science  of  pedagogy  (Dewey, 
1929b) 

Summary.  Effect  size  of  biomedical  science  achievement  is  a  function  of 
assessment  type  independent  of  whether  contrasts  between  lecture-based  and  problem- 
based  instruction  were  conducted  using  historical  controls  at  the  same  medical  school, 
between  medical  schools,  or  within  medical  schools  between  parallel  curricula.  Present 
findings  suggest  that  measures  of  differential  instruction  effect  are  most  sensitive  when 
assessment  is  aligned  with  the  specific  objectives  of  instruction.  External  assessment  is 
too  global  to  reveal  functional  infomiation  regarding  instructional  effect.  Because 
standardized  assessments  are  antithetical  to  the  goals  of  problem-based  teaching  and 
learning,  they  may  constrain,  if  not  restrain,  desired  instrtictional  outcomes.  Indeed, 
present  findings  suggest  that  when  assessments  are  congnient  with  educational 
objectives,  the  relative  effects  of  problem-based  instruction  emerge.  As  argued  earlier,  1 
suspect  the  results  of  standardized  achievement  reflect  the  inadequacy  of  assessment 
more  than  they  do  the  inadequacy  of  problem-based  teaching  and  learning.  However,  the 
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effect  of  licensing  examinations  may  be  so  powerful  as  to  mask  differential  learning 
methods.  Particular  evidence  for  this  claim  is  provided  by  virtually  equivalent  licensing 
examination  performance  by  students  undergoing  medical  education  at  the  same 
mstitution  in  a  lecture-based,  problem-based,  or  independent  study  curriculum  path  (Way 
et  al.,  1999).  More  importantly,  if  licensing  examinations  are  the  most  significant 
influence  in  medical  student  learning,  then  future  studies  should  examine  differences 
among  examination  items,  particularly  since,  as  mentioned  earlier,  findings  of  this 
research  suggest  that  the  objectives  driving  the  conversion  from  NBME  Part  I  to  USMLE 
Step  1  have  not  been  realized.  Rote  memorization  still  appears  to  be  a  successful  strategy 
for  passing  USiVILE  Step  1 .  Apparently,  in  an  "acute  learning  context,"  medical  students 
tend  to  learn  for  examinations,  not  practice.  Clearly,  this  is  not  a  desirable  outcome 
under  any  circumstance.  Although  performance  on  USMLE  Step  1  is  considered  an 
important  gauge  of  biomedical  science  knowledge,  passing  licensing  examinations  should 
be  defaclo.  Since  commercial  review  books  and  courses  are  available  to  facilitate  study 
for  licensing  examination,  teaching  and  learning  should  be  aligned  with  the  cognitive  and 
noncognitive  aims  of  medical  education  rather  than  oriented  toward  knowledge  assessed 
by  licensing  examination  boards,  hideed,  present  findings  confirm  in  their  present 
configuration  that  standardized  tests  lack  the  sensitivity  necessary  to  differentiate  the 
effects  of  instruction.  Valid  assessment  of  pedagogy  and  the  learning  process  requires  a 
variety  of  strategies  that  capture  appropriate  knowledge  and  skills,  that  is,  multi-method 
assessments.  Especially  as  discussed  in  Chapter  2,  if  a  "compensation  hypothesis,"  as 
advanced  by  Richards  and  Cariaga  (1993)  operates  in  PBL  students  with  respect  to 
licensing  examinations,  educators  face  a  dilemma.  Since  licensing  examinations  are 
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permanent  elements  of  the  academic  landscape,  a  challenge  for  problem-based  instniction 
designers  is  to  blimt,  or  obviate  the  student's  need  to  compensate  for  a  perceived  lack  of 
factual  knowledge,  while  facilitating  the  license  examination  success. 

Research  question  3.  Is  there  a  significantly  positive  effect  size  at  the  .05  level 
for  clinical  science  achieveinent  in  students  and  physicians  who  undergo  medical 
education  with  problem-based  compared  to  lecture-based  teaching  and  learning? 
Analysis  of  twenty-nine  independent  samples  involving  7,340  subjects  revealed  a 
significant  effect  size  for  clinical  science  achievement  (ES  =  0.32,  95%C1  0.27  to  0.38) 
favoring  students  and  physicians  who  undergo  problem-based  instruction.  Sensitivity 
analysis  revealed  inclusion  of  the  only  outlier  (ES  =  1 .76)  would  increase  effect  size  by 
21%,  but  does  not  materially  change  the  results  or  conclusions  drawn  from  the  results. 
The  impressive  effect  of  the  outlier  is  augmented  by  weighting  (n  =  641).  Analyzed  via 
narrative  in  Chapter  4,  the  outlier  study  contrasted  students  from  Maastricht  University, 
undergoing  what  is  arguably  the  most  student-centered  version  of  re-iterative  PBL  in 
published  literature,  compared  to  students  undergoing  traditional  lecture-based, 
discipline-oriented  instruction  at  two  other  institutions.  As  discussed  in  Chapter  2, 
Maastricht  University's  problem-based  teaching  and  learning  ("seven-jump")  and 
assessment  ("triple-jump")  approaches  are  uniquely  consonant  with  some  principles  of 
infomiation  processing,  social  constructivism,  and  situated  learning,  and  was  discussed  in 
Chapter  2.  Of  twenty-nine  non-outlier  effect  sizes,  86%  (n  =  25)  were  positive  and  45% 
(n  =  13)  were  significantly  positive.  Only  one  effect  size  estimate  involving  less  than 
one-percent  of  participants  (n  =  53)  was  significantly  negative.   From  a  quantitative 
perspective,  problem-based  instruction  under  average  conditions  represented  by  these 
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studies  can  be  expected  to  move  a  typical  learner  from  the  50""  to  the  63"'  percentile  of 
the  lecture-based  instruction  population.  Robustness  of  this  data  set  is  supported  because 
a  "fail-safe  N"  estimate  {Rosenthal,  1979)  would  require  1,209  additional  null  studies  to 
reduce  the  significance  ofclinical  knowledge  achievement  to  less  than  .05.  In  addition, 
there  was  no  evidence  of  publication  bias  (Spearman's  rho  =  -.13,  p  =  .52). 

Research  question  4.  Is  variation  in  aggregated  effect  size  estimates  for  clinical 
science  achievement  greater  than  the  sampling  error  warranting  subgroup  analyses  to  test 
for  potential  moderation  by  a  priori  defined  outcome  measures,  methodological  features, 
and  substantive  features  of  problem-based  teaching  and  learning?  Variance  analysis 
suggests  that  the  distribution  of  effect  size  estimates  for  clinical  science  achievement  is 
not  homogeneous  (Qh  =  162.8,  p  <  .00001 );  this  is  greater  than  would  be  expected 
entirely  from  sampling  error.  A  priori  conditional  moderators  ofclinical  science 
achievement,  including  achievement  measure,  methodological  features  and  substanfive 
characteristics,  were  examined  with  analysis  of  variance  modeling.  Cumulative  findings 
of  current  research  reveal  that  instructional  effects  for  clinical  achievement  differentiated 
with  licensing  examination  data  significantly  favored  problem-based  teaching  and 
learning,  ES  =  0.24,  95%CI  0.17  to  0.31.  Vernon  and  Blake  (1993)  found  a  mean  effect 
size  of  0.08  {95%CI  -0.05  to  0.21)  for  licensing  measures  (n  =  3,  all  NBME  Part  II)  of 
clinical  achievement.  Before  the  shift  from  testing  primarily  factual  and  abstract 
knowledge  (i.e.,  NBME  Part  II)  to  incorporating  knowledge  application  assessment  (i.e., 
USMLE  Step  2),  few  investigafions  reported  significant  differences  in  examinafion 
scores  between  students  who  underwent  problem-based  and  lecture-based  instruction. 
Mobilizing  a  similar  argument  to  the  one  employed  earlier  regarding  USMLE  Step  I, 
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and  assuming  that  problem-based  learning  facilitates  higher-order  thinking  (e.g.,  the 
application  of  integrated  biomedical  and  clinical  science)  and  that  the  academic 
characteristics  of  students  have  remained  relatively  stable,  this  finding  is  important 
because  it  provides  some  evidence  of  constnict  validity  for  the  USMLE  Step  2 
examination.  The  differential  effect  of  problem-based  versus  lecture-based  instruction 
contrasted  with  NBME  Part  11  (ES  =  0.12,  95%C1  0.01  to  0.22)  versus  USMLE  Step  2 
(ES  =  0.39,  95%CI  0.28  to  0.49)  is  significantly  differem. 

Analogous  to  analyses  of  biomedical  science  achievement,  effect  size  for  clinical 
science  achievement  was  greater  when  investigators  compared  the  relative  effects  of 
problem-based  and  lectured  based  instruction  with  non-licensing  assessments,  ES  =  0.37, 
95%CI  0.30  to  0.44.  However,  unlike  biomedical  science  achievement,  clinical  science 
achievement  measured  with  both  licensing  (ES  =  0.24,  95%C1  0.17  to  0.31)  and  non- 
licensing  assessments  (ES  =  0.37,  95%C1  0.30  to  0.44)  significantly  favored  learners  who 
underwent  problem-based  instruction.  Effect  size  was  significantly  greater  when 
instructional  impact  was  quantified  with  non-licensing  assessments  (Qb  =  6.9,  p  <  .001). 
Of  non-licensing  assessments,  effect  sizes  were  highest  when  researchers  employed 
locally  constructed,  criterion-based  multi-method  (ES  =  0.48)  and  multiple-choice 
question  (ES  =  0.46)  assessments,  and  NBME-11  Shelf-boards  (ES  =  0.45).  They  were 
lowest  when  supervisor  ratings  of  clinical  achievement  were  contrasted  (ES  =  0.29). 
Unfortunately,  no  reports  contrasting  supervisor  ratings  provided  inter-rater  reliability 
and  within-group  variance  was  significant.  Thus,  to  examine  methodological  and 
substantive  study  features  as  potential  moderators  of  instructional  effect,  further  analyses 
of  non-licensing  measures  were  conducted  to  discover  grouping  patterns  with  relatively 
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stable  variance  and  homogeneity.  Sub-grouping  independent  samples  and  comparing 
scores  from  NBME-ll  Shelf-boards,  multimethod  and  MCQ  assessments  produced 
statistically  comparable  effect  size  estimates.  Sub-grouping  reduced  heterogeneity  by 
65%.  Subsequent  methodological  and  substantive  features  analyses  assumed  sufficient 
variance  was  explained  by  the  characteristics  of  this  relatively  homogeneous  sub- 
grouping.  Evidence  from  this  reduced  data  set  with  an  effect  size  of  0.47  (95%CI  0.38  to 
0.56)  suggests  problem-based  instruction  would  shift  clinical  science  achievement  in  a 
student  undergoing  lecture-based  instruction  from  the  50"'  percentile  to  the  68"" 
percentile.  The  robustness  of  the  reduced  data  set  is  supported  because  a  "fail-safe  N" 
estimate  (Rosenthal,  1979)  would  require  454  additional  null  studies  to  reduce  the 
significance  of  clinical  knowledge  achievement  to  less  than  .05.  There  is  no  evidence  of 
publication  bias  for  this  reduced  sample,  however,  external  validity  may  be  compromised 
via  reactive  or  interaction  effect  of  assessment  (Campbell  &  Stanley,  1966).  Thus, 
caution  is  urged  when  interpreting  results  and/or  drawing  conclusions  derived  from  this 
reduced  data  set. 

The  mean  effect  size  estimate  for  clinical  science  achievement  was  significantly 
greater  when  sampling  included  intact  cohorts  (ES  =  0.5 1 )  versus  random  assignment  (ES 
=  0.29).  No  other  methodological  feature  significantly  moderated  effect  size  estimates. 
Similar  to  biomedical  science  achievement,  the  scope  of  problem-based  instruction  was 
not  a  significant  moderator  of  clinical  science  achievement.  However,  other  substantive 
features  significantly  moderated  the  clinical  science  achievement  effect  of  problem-based 
teaching  and  learning  including  PBL  taxonomy,  problem-type,  tutor  expertise,  and 
responsibility  for  self-directed  learning.  Reiterative  problem-based  instruction  and  a 
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general  problem-oriented  approach  affected  similar  significant  effect  sizes  and  were 
much  greater  than  those  produced  using  a  modified  case-method.  A  paucity  of  details 
reported  in  primary  research  regarding  the  implementation  of  PBL  not  operationalized  as 
reiterative  renders  the  significance  of  this  finding  unclear.  By  design,  reiterative  PBL 
focuses  student  generated  learning  issues  to  guide  self-directed  learning  and  the 
subsequent  sharing  of  gained  knowledge.  It  would  be  important  to  know  if  coupling  self- 
directed  and  collaborative  learning  in  reiterative  fashion  offers  no  substantive  benefit. 
Logistically,  reiterative  PBL  increases  the  temporal  requirement  for  resolving  learning 
objecUves  stimulated  by  problem  challenge.  If  achievement  is  not  superior,  then 
reiterative  PBL  may  prove  impractical  vis-a-vis  time  and  expense.  Although 
theoretically  and  logically  sound.  Barrows  claim  (1986,  2000)  that  reiterative  is  uniquely 
superior  to  other  species  of  problem-based  teaching  and  learning,  is  not  supported  by  the 
available  evidence  examined  in  present  research. 

With  support  from  information  processing  theory  (Chi  &  Bassok,  1989;  Chi  et  al., 
1994;  Simon,  1973),  advocates  of  problem-based  instruction  claim  ill-structured 
problems  are  required  to  facilitate  free-inquiry  and  unconstrained  student  generated 
learning  issues  (Barrows,  1986,  2000).  The  findings,  however,  support  no  unique 
superiority  afforded  by  a  particular  problem  structure.  Problems  posed  in  both  ill- 
structured  and  well-structured  fomiats  produced  equally  the  largest  effect  sizes,  although 
the  confidence  intervals  associated  with  effect  sizes  for  ill-structured,  well-structured,  and 
mixed  structured  problem  type  fomiats  overlapped.  Additional  research  is  required  to 
inform  instmctional  designers  regarding  the  situational  value  and  unique  superiority  of 
problem  structure  as  a  function  of  desired  learning  objectives.   In  particular,  it  would  be 
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important  to  know  whether  problem-strticture  efficacy  varies  with  educational  objectives, 
context,  and  assessment  instrument. 

Possibly  the  integration  of  biomedical  and  clinical  science  knowledge,  regardless 
of  problem  structure,  dominates  the  observed  achievement  effect.  Current  research 
suggests  that  independent  of  espoused  species  of  problem-based  instruction,  the 
integration  of  biomedical  and  clinical  science  knowledge  appears  ubiquitous  among 
varied  implementations  of  problem-based  teaching  and  learning.  Clearly,  pedagogical 
methods  facilitating  the  proceduralizing  of  declarative  knowledge  (Anderson,  1990)  or 
coupling  concepts  and  results  (Brown  et  al.,  1989;  Cobb,  2002;  Perkins,  1986)  are  viable 
candidates  for  uniquely  efficacious  ingredients  of  problem-based  teaching  and  learning. 

Present  findings  revealed  no  differential  effect  when  responsibility  for  generating 
learning  issues  resided  with  students  or  a  combination  of  students  and  faculty.  Since 
problem-structure  and  origin  of  learning  issues  did  not  significantly  moderate  effect  size, 
the  practical  utility  of  claims  to  the  contrary  remains  to  be  substantiated  by  future 
process-oriented  research. 

Content  expert  versus  peer  tutors  was  associated  with  significantly  greater  clinical 
science  achievement.  No  reports  cited  use  of  non-content  expert  faculty  tutors,  and  only 
two  of  twelve  reports  involved  peer-tutors.  Effect  size  was  greatest  when  responsibility 
for  self-directed  learning  was  directed  by  students  supplemented  with  mandatory  lectures, 
but  was  not  significantly  different  than  when  self-directed  learning  was  guided 
exclusively  by  students,  or  when  supplemented  with  elective  lectures.  Considered 
together,  these  results  suggest  that  most  problem-based  tutorial  groups  are  guided  by 
content  expert  tutors  and  whether  a  student  is  solely  responsible  or  not  for  self-directed 
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learning  does  not  significantly  affect  clinical  achievement,  which  is  counter  to  the  claims 
of  reiterative-PBL  advocates  (Barrows,  1986,  2000).  These  findings  are  supported  by 
research  suggesting  an  inverse  relationship  between  tutor  expertise  and  self  directed 
learning  (Schmidt  &  Moust,  1995).  Consequently,  content  expert  tutors  may  foster 
achievement  independent  of  whether  students  can  or  cannot  exercise  ownership  of 
learning  objectives. 

Summary.  The  present  findings  for  clinical  science  achievement  do  not  support 
claims  regarding  the  uniquely  superior  substantive  dimensions  of  problem-based 
learning.  Evidence  from  this  research  suggests  efficacy  of  problem-based  versus 
traditional  lecture-based  teaching  and  learning  for  facilitating  superior  clinical  science 
achievement  is  independent  of  implementation  method.  This  instructional  benefit  likely 
accrues  from  the  integration  of  biomedical  and  clinical  science  in  the  context  of  a  health 
care  problem.  This  conclusion  is  consistent  with  some  perspectives  of  social 
constructivism  (e.g.,  pragmatic  construction,  Cobb,  2002)  and  situational  learning  (e.g., 
theory  and  practice  interaction.  Brown  et  al,  1989;  Greeno,  1998),  as  discussed  in 
Chapter  2.  Further  evidence  supporting  the  integration  of  biomedical  and  clinical  science 
as  a  uniquely  superior  feature  of  problem-based  teaching  and  learning  is  found  in  more 
process-oriented  research  specifically  examining  problem-solving  skills.  These  findings 
are  discussed  next. 

Research  question  5.  Is  there  a  significant  positive  effect  size  at  the  .05  level  for 
problem  solving  achievement  in  students  and  physicians  who  undergo  medical  education 
with  problem-based  compared  to  lecture-based  teaching  and  learning?  An  analysis  often 
independent  studies  involving  1,236  participants  revealed  significant  effect  size  estimates 
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for  problem-solving  skills  (ES  =  0.19,  95%C1  0.06  to  0.32)  benefiting  students  and 
physicians  who  iindei-went  problem-based  instniction.  All  ten  effect  size  estimates  were 
positive  and  40%  were  significant.  Effect  size  estimated  from  this  data  set  suggests  an 
average  student  would  move  from  the  50'*'  percentile  to  the  58"'  percentile  had  they 
undergone  problem-based  versus  lecture-based  instniction.  The  fail-safe  calculation  for 
this  data  set  is  60  (Rosenthal,  1979);  robustness  is  marginal  because  only  81  additional 
studies  with  a  mean  effect  size  ofzero  are  required  to  reduce  the  combined  significance 
of  problem-solving  achievement  to  less  than  .05. 

Evidence  from  the  present  study  suggests  that  instructional  method  does  influence 
problem-solving  strategies  and  solution  accuracy  in  medical  students.  Differential 
instructional  effect  vis-a-vis  medical  problem-solving  success  appears  not  to  be  due  to 
knowledge  accumulation,  but  to  increased  elaboration  of  knowledge.  Findings  indicate 
that  problem-based  versus  traditional  lecture-based  teaching  and  learning  enables 
students  to  better  apply  biomedical  science  concepts  to  novel  clinical  problems  and  to 
exhibit  superior  success  in  resolving  health  care  problems.  For  example,  Schmidt  and 
associates  (1996)  observed  a  mean  differential  problem  solving  accuracy  of  1.5  out  of  30 
epidemiologic  relevant  clinical  cases  between  senior  medical  students  who  underwent 
lecture-based  versus  problem-based  curricula.  If  teaching  and  learning  effect  translates  to 
practice,  a  differential  performance  of  misdiagnosing  1 .5  patients  per  30  patient  visits 
would  not  be  trivial  in  terms  of  health  care  efficacy  and  financial  expense.  Additional 
insight  for  medical  educators  emerges  from  a  third  group  studied  by  Schmidt  and 
associates.  This  group  underwent  a  lecture-based  curriculum,  but  with  an  integrated 
approach  (didactics  integrating  biomedical  and  clinical  science).  In  this  well-matched 
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group,  there  were  no  differences  in  problem  solving  (diagnostic)  accuracy  with  students 
from  a  problem-based  curriculum.  Discovering  problem-solving  success  parity  between 
problem-based  instruction  and  lecture-based,  integrated  curriculum  is  an  important 
finding  with  significant  implications  for  medical  instrucUon.  In  particular,  the  authors 
suggest  that  the  integration  of  biomedical  and  clinical  science  grounded  in  health  care 
problems  is  probably  an  essential  feature  that  enables  superior  clinical  problem  solving 
perfomiance  independent  of  instructional  method,  whether  lecture-based  or  tutorial 
group-based.  If  relevant  content  retention  and  problem  solving  skills  are  dominant 
educational  goals,  then  the  integrafion  of  biomedical  and  clinical  science  via  a  lecture- 
based  instructional  format  may  be  efficacious.  Indeed,  unequivocal  implications  of  this 
research  are  that  teaching  and  learning  with  versus  without  problein  posing  as  the  locus 
of  instruction  with  integration  of  biomedical  and  clinical  science  facilitates  superior 
problem-solving  skills  in  medical  students.  Presumably,  the  integration  of  theoretical  or 
abstract  concepts  (biomedical  science)  and  practice  (clinical  science)  facilitates  the 
constniction  of  concept-tools  or  "illness-scripts,"  as  was  discussed  in  Chapter  2. 

Summary.  The  present  findings  are  consistent  with  the  information-processing 
theory  that  predicts  domain  knowledge  mastery  is  necessary,  but  not  sufficient  for 
problem-solving  success  (Anderson,  1993,  1996;  Newell  &  Simon,  1972).  Problem 
solving  ability  is  infiuenced  not  just  by  quantity  of  knowledge,  but  by  the  extent  to  which 
knowledge  is  organized  into  coherent  networks  of  biomedical  science  concepts  and 
clinical-disease  categories  (e.g.,  illness-scripts,  Schmidt  et  al.,  1990).  The  present  results 
suggest  that  students  who  undergo  lecture-based,  discipline-oriented  instruction  are 
competent  reciters  of  facts,  but  much  less  successful  problem  solvers.  Physician 
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competency  is  judged  by  the  ability  to  discover,  operatioiialize,  and  take  action  to  solve 
clinical  problems,  a  disposition  described  by  Schon  (1987)  as  knowing-in-action.  For 
medical  education,  a  pedagogical  imperative  is  to  teach  students  to  be  competent 
practicing  on  the  fringe  of  surprise  in  clinical  problem  spaces.  Understanding  the 
uniqueness  and  uncertainty  of  authentic  problems  can  serve  as  a  foundation  for  practice 
competence  (Dewey  1933/1998;  Gibson,  1977;  Greeno,  1998;  Lave  &  Wegner,  1991; 
Schon).  According  to  some  proponents  of  situated  learning,  these  necessary  skills  are 
better  learned  by  solving  problems  in  a  setting  that  facilitates  immediate  feedback, 
reinforcement,  and  correction  (Brown,  1994;  Brown  et  al.,  1989).  Cumulative  evidence 
analyzed  for  the  present  study  suggests  differential  orientation  to  medical  knowledge 
acquisition  exhibited  by  problem-based  verses  lecture-based  learners  effects  differences 
in  problem-solving  strategies.  Consequently,  two  distinct  solution  patterns  become 
evident.  The  orientation  of  PEL  students  is  characterized  by  the  extensive  use  of 
situational  infomiation  (i.e.,  problem-space  affordances,  Greeno)  and  an  aim  to  construct 
a  cumulative  interpretation  of  a  patient's  pathophysiological  condition  (Cobb,  2002; 
Resnick,  1989,  1991)  using  relevant  biomedical  science  concepts.  Lecture-based 
students  tend  to  be  guided  by  objective  orientation  characterized  by  a  reactive 
disposition,  which  appears  grounded  in  deterministic  implementation  of  initial  hypothesis 
(Lmdblom-Ylanne  &  Lonka,  2002).  That  is,  pre-conceived  notions  guide  observational 
perspective  (constrained  perception)  and  frame  interpretation  of  problem-space 
affordances  (Gibson).  The  significance  of  these  observed  outcomes  for  professional 
practice  remains  to  be  discovered.  Clearly,  a  one-dimensional  (reactive)  orientation  to  a 
complex  health  care  problem  is  inadequate. 
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Research  question  6.  Is  variation  in  aggregated  effect  size  estimates  for  problem 
solving  achievement  greater  than  the  sampling  error  warranting  sub-group  analyses  to 
test  for  potential  moderation  by  a  priori  defined  outcome  measures,  methodological 
features,  and  substantive  features  of  problem-based  teaching  and  learning?  Problem- 
solving  effect  sizes  were  homogeneous  (Q  =  8.1,  p  =  .53),  thus  suggesting  their 
distribution  can  reasonably  be  attributed  to  sampling  error  alone.  Because  there  is  no 
conditional  variance,  the  mean  of  effect  size  estimates  is  considered  an  accurate 
reflection  of  the  comparative  advantage  of  problem-based  versed  lecture-based 
instruction  for  promoting  problem-solving  success  (Hedges,  1 994). 

Research  question  7.  Is  there  a  significant  positive  effect  size  at  the  .05  level  for 
self-directed  learning  in  students  and  physicians  who  undergo  medical  education  with 
problem-based  compared  to  lecture-based  teaching  and  learning?  Analysis  of  nineteen 
independent  samples  involving  3,487  participants  showed  a  significant  effect  size  for 
self-directed  learning  skills  (ES  =  0.47,  95%CI  0.39  to  0.55)  advantaging  students  and 
physicians  who  undergo  problem-based  instruction.  All  nineteen  effect  sizes  were 
posifive  and  eleven  (58%)  were  significant.  These  findings  suggest  problem-based 
instrucdon  under  average  conditions  would  be  expected  to  transition  a  typical  learner's 
self-directed  learning  skills  from  the  50"^  percentile  to  the  68"'  percentile  in  the  lecture- 
based  instruction  population.  The  robustness  of  the  main  effect  size  estimate  is 
substantiated  because  1,013  additional  studies  with  a  mean  effect  size  of  zero  are  required 
to  reduce  the  combined  significance  of  clinical  achievement  outcomes  to  less  than  .05 
(Rosenthal,  1979).  In  addition,  the  likelihood  of  publication  bias  is  remote  (Spearman's 
rho  =  -.05,E  =  .83). 
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In  addition  to  varied  resource  utilization  and  study  patterns,  results  of  the  current 
analysis  suggests  that  students  undergoing  problem-based  versus  lecture-based 
instruction  are  more  likely  to  use  self-directed  learning  for  meaning  (Newble  &  Clark, 
1986;  van  Berkel,  Nuy  &  Geerligs,  1995)  and  are  less  likely  to  study  for  reproduction  of 
information  (Lindblom-Yliinne  &  Lonka,  1999;  Regan-Smith  ct  al.,  1994).  These 
findings  are  consistent  with  principles  proposed  by  some  social  constructivists  (Bereiter, 
1994;  Bruning  et  al.,  1999;  Cobb,  2002).  In  this  research,  constructivism  is  a  learning 
process  that  assumes  opportunities  are  provided  to  personally  make  knowledge 
representations,  contrast  one's  understanding  of  that  knowledge  with  others' 
understanding  (Bereiter;  Cobb;  Prawat  &  Floden,  1994),  and  refine  or  restructure 
knowledge  as  a  more  relevant  experience  (Cobb),  as  was  discussed  in  Chapter  2. 
What  evidence  is  there  to  suggest  that  self-directed  learning  is  not  just  a 
coincidental  reaction  to  instructional  approach  but  does  persist?  Unfortunately,  this 
question  remains  unanswered  because  only  six  of  nineteen  studies  contrasted  self- 
directed  learning  outcomes  in  physicians  in  training  or  practice  who  had  undergone 
problem-based  versus  lecture-based  medical  education.  In  this  subgroup,  an  effect  size  of 
0.20  was  significant  because  95%CI  0.08  to  0.33  did  not  include  zero.  However,  varied 
measures  of  self-directed  learning  were  employed  and  the  grouping  was  heterogeneous 
(Qwi  =  20.35,  p  =  .001).  Moreover,  two  of  six  studies  reported  negative  effect  sizes  for  a 
mean  of -.08.  If  self-directed  learning  is  transient,  research  is  required  to  examine  what 
can  be  done  during  clerkship  years  and  residency  training  to  sustain  the  self-directed 
learning  fostered  during  problem-based  teaching  and  learning. 
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Becoming  an  expert  learner  may  be  the  most  powerful  outcome  of  formal  medical 
education  (Schon  1987).  Developing  a  disposition  for  life-long  learning  is  the  rationale 
for  fostering  self-directed  learning  (Barrows,  1 986,  2000;  Bereiter  &  Scardamalia,  2000; 
Mann,  1994).  Possibly  the  most  important  element  in  lifelong  learning  is  attitude  toward 
infonnation  acquisition  and  understanding  (i.e.,  sensitivity  to  evidence);  this  is  discussed 
later.  If  a  student  values  information  as  a  means  to  improve  clinical  practice,  a  habit  of 
acquiring  and  using  information  will  develop  rapidly.  However,  as  discussed  in  Chapters 
1  and  2,  empirical  and  anecdotal  evidence  suggest  that,  independent  of  instructional 
method,  information  is  acquired  to  support  an  educational  process  in  general  and 
assessments  in  particular  ("hidden  curriculum").  Designing  assessments  that  correspond 
with  eventual  practice  patterns  appears  crucial  to  afford  students  oppoilunities  to  value 
and  develop  applicable  information  gathering  and  learning  skills  (pragmatic 
constructivism,  Cobb,  2002).  Lecture-based,  knowledge-intensive  instruction  often  is 
assessed  with  objective  examinations  incoiporating  primarily  multiple-choice  questions. 
Such  examinations  typically  capture  a  student's  capacity  to  memorize  content  rather  than 
measure  capability  for  rigorously  analyzing,  synthesizing,  evaluating  and  applying 
knowledge  (Lindblom-Ylanne  &  Lonka,  1999,  2002).  Extending  Dewey's  (1933/1998) 
concept  of  reflective  inquiry,  Schon  (1987)  recommended  medical  students  be  coached  in 
a  "reflective  practicum"  (p.  157),  thus  acquiring  knowledge  and  skills  via  self-directed 
learning  to  perform  in  undetcmiined  zones  of  medical  practice.  The  evidence  suggests 
that  problem-based  teaching  and  learning  catalyzes  an  indeterminate  situation,  and  invites 
students  to  engage  problems  in  a  process  of  reflective  inquiry  and  self-directed  learning 
(Dewey)  and  reflection-in-action  (Schon).  However,  there  is  little  evidence  linking  self- 
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directed  learning  stimulated  by  instructional  method  and  a  disposition  toward  life-long 
learning.  Whether  PBL  prepares  students  to  engage  in  learning  activities  that  are  self- 
defined  rather  than  defined  by  a  teacher  is  unclear  and  warrants  further  research. 

Summary.  Cultivating  a  durable  curiosity  in  conjunction  with  the  ability  to 
continuously  self-audit  practice  patterns  and  self-direct  appropriate  learning  appear  to  be 
appropriate  educational  imperatives.  Evidence  for  methods  to  accomplish  this 
pedagogical  challenge  remains  illusive.  Rofiective  inquiry  appears  to  be  a  suitable 
methodological  strategy  for  knowmg,  analyzing,  comparing  and  contrasting  concepts  and 
their  implications  for  practice.  Moreover,  Dewey's  (1933/1998)  method  of  thinking  and 
learning  may  facilitate  "deep  cognitive  processing"  (Coles,  1990,  1997;  Craik  & 
Lockhart,  1972;  Newble  &  Clarke,  1986).  An  expressed  aim  of  medical  educafion  is  to 
enable  medical  students  and  physicians  to  assemble  relevant  infomiation,  analyze      . 
infonnation,  improve  their  problem-solving  skills  and  refiect  on  their  learning  process 
(Liaison  Committee  on  Medical  Education,  2002).  Although  theoretically  attractive, 
whether  fostering  development  of  a  disposition  for  reflective  inquiry  will  develop  and 
sustain  efficacious  self-directed  learning  is  an  important  question  and  may  be  a  valuable 
area  for  research.  Nurturing  in  students  a  disposition  incorporating  Dewey's  method  of 
refiective  inquiry  to  construct  warranted  assertions  very  well  may  be  the  elixir  for 
medical  reasoning  in  general,  and  self-directed  learning  in  particular.  Possibly  the  most 
important  outcome  of  self-directed  learning  is  the  development  of  practice  patterns  that 
are  truly  self-  versus  other-directed.  Clearly,  this  shift  from  relying  on  a  "syllabus,"  to 
relymg  on  self,  requires  practice  and  a  performance  history  of  success  (Lovett  & 
Anderson,  1996)  or  "collection  of  sufficient  instances"  (Dewey,  1933/1998,  p.  173). 
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Research  question  8.  Is  variation  in  aggregated  effect  size  estimates  for  self- 
directed  learning  greater  than  the  sampling  error  warranting  sub-group  analyses  to  test  for 
potential  moderation  by  a  priori  defined  outcome  measures,  methodological  features,  and 
substantive  features  of  problem-based  teaching  and  learning?  Variance  analysis  suggests 
the  distribution  of  effect  size  estimates  for  self-directed  learning  is  not  homogeneous  and 
greater  than  would  be  expected  entirely  from  sampling  error  (Qb  =1 1 1 .8,  p  <  .001  )■ 
Thus,  the  null  hypothesis  is  rejected  that  sampling  error  alone  explains  this  magnitude  of 
variance  in  self-directed  learning  effect  sizes.  This  finding  suggests  further  exploration  is 
warranted  to  discover  whether  outcome,  methodological  or  substantive  study  features  are 
operating  to  moderate  variance  of  effect  sizes. 

Effect  size  estimates  derived  from  varied  measures  of  self-directed  learning, 
including  resource  utilization,  study  time,  learning  process,  and  practice  competence, 
were  statistically  equivalent,  Qij(3)  =  3.2,  p  <  .36.  Consequently,  the  conditional  variance 
of  effect  size  estimates  appears  to  be  well  explained  by  type  of  outcome  measure.  Thus, 
variance  analysis  of  methodological  and  stibstantive  features  to  discover  potential 
modifiers  of  effect  size  magnitude  is  statistically  inappropriate. 

Research  question  9.  Is  there  a  significant  positive  effect  size  at  the  .05  level  for 
attitude  toward  learning  in  students  and  physicians  who  undergo  medical  education  with 
problem-based  coinpared  to  lecture-based  teaching  and  learning?  An  analysis  of  thirty 
independent  studies  involving  5,670  participants  suggests  significant  effect  size  for 
positive  attitudes  toward  learning  (ES  =  0.45,  95%C1  0.39  to  0.52)  favoring  medical 
students  and  physicians  who  underwent  problem-based  instruction.   All  thirty  effect  size 
estimates  were  positive  and  seventeen  effect  sizes  were  significant.  The  findings  suggest 
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that  problem-based  instruction  is  expected  to  improve  attitude  toward  learning  for  the 
average  student  undergoing  lecture-based  instruction  from  the  SO"'  percentile  to  the  69"' 
percentile.  This  main  effect  is  robust  because  2,5 1 8  additional  studies  with  a  mean  effect 
size  of  zero  are  required  to  reduce  the  combined  significance  of  attitude  toward  learning 
outcomes  to  less  than  .05  (Rosenthal,  1979).  Publication  bias  for  this  data  set  is  unlikely 
because  there  was  no  significant  correlation  between  effect  size  estimates  and  sample  size 
(Spearman's  rho  =  -.23,  p  =  .22). 

Attitude  toward  learning  outcomes  were  defined  as  perceptions  of  positive 
learning  environmem,  utility  of  education  and  level  of  academic  anxiety.  Taken  together, 
evidence  for  the  positive  effect  of  learning  environment  and  utility  of  education  suggests 
problem-based  learning  projects  face  validity.  Face  validity  of  educational  experience  is 
most  likely  a  strong  motivational  factor  (Mosier,  1947  cited  in  Crocker  &  Algina,  1986). 
Moreover,  learning  activities  perceived  as  valid  probably  are  perceived  as  efficacious 
(Brown  et  al.,  1989;  Cobb,  2002).  Learning  environments  with  inherem  validity  and 
efficacy  provide  boundless  possibilities  for  knowledge  construction,  application, 
synthesis,  evaluation  and  re-construction  (Cobb;  Savery  &  Duffy,  1995).  Although 
operationalized  separately  in  this  research,  clearly  perceived  validity  and  efficacy  of 
learning  environment  directly  influences  achievement,  problem-solving  skills,  and 
capability  for  self-directed  learning.  As  discussed  earlier,  to  predict  effectively  the 
transaction  between  studem  and  learning  environment,  it  is  essential  for  educators  to 
verify  conditions  of  learning  environment,  both  actual  and  perceived.  The  art  of  medical 
education  should  imitate  the  life  of  clinical  practice  (Dewey,  1948,  1958;  Schon  1987). 
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Research  question  10.  Is  variation  in  aggregated  effect  size  estimates  for  altitude 
toward  learning  greater  than  the  sampling  error  waiTanting  sub-group  analyses  to  test  for 
potential  moderation  by  a  priori  defined  outcome  measures,  methodological  features,  and 
substantive  features  of  problem-based  teaching  and  learning?  Analyses  of  variance 
suggest  distribution  of  effect  size  estimates  for  attitude  toward  learning  is  not 
homogeneous  and  greater  than  would  be  expected  entirely  from  sampling  error  (Q 
=138.9,  g<  .00001).  Consequently,  theoretically  and  empirically  derived  outcome 
measures,  methodological  and  substantive  variables  of  problem-based  teaching  and 
learning  were  examined  as  potential  moderators  of  positive  attitude  toward  learning. 
Neither  source  nor  type  of  measure  employed  to  quantify  perceived  attitude  toward 
learning  operationalized  as  positive  learning  environment,  utility  of  education,  or 
academic  anxiety  significantly  moderated  effect  size  estimates.  Including  studies  with 
diverse  reliability  and  validity  considerations  did  not  significantly  influence  effect  size 
estimates.  Indeed,  sensitivity  analysis  revealed  the  exclusion  of  data  measured  with 
instruments  in  which  validity  was  not  reported  does  not  significantly  alter  findings,  but 
would  reduce  the  data  set  by  52  percent.  Methodological  moderators  include  number  of 
involved  study  sites,  response  rate,  geographical  site  of  research,  and  publication  decade. 
Studies  conducted  in  multiple  centers  (25%  of  studies)  reported  less  effect,  but  the 
difference  was  not  significant.  Mean  effect  size  estimate  was  highest  when  response  rate 
was  not  reported  (17%  of  studies),  however,  there  were  no  significant  differences  in 
effects  when  response  rates  were  stratified  in  decades  from  100%  to  less  than  70%,  or  not 
reported.  Research  was  conducted  in  six  countries  or  geographic  regions  with  40%  of 
reports  irom  the  United  States.  The  effect  of  problem-based  instruction  was  greatest  in 
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Scandinavia  {ES  =  .94,  1 0%  of  studies)  and  this  difference  is  significant  for  every  site 
except  Gennany.  There  are  no  effect  size  differences  among  reports  from  the  United 
States,  Canada,  Australia,  the  United  Kingdom  and  Germany.  Other  than  length  of 
medical  school  and  potential  variation  in  student  characteristics,  there  were  no  apparent 
substantive  features  distinguishing  instructional  approaches  employed  in  Scandinavian 
mvestigations.  Whether  student  characteristics  played  a  role  is  speculative.  Most 
students  enter  Scandinavian  medical  school  immediately  from  high  school  and  the 
curriculum  is  five  or  six  years.  This  distinction  separates  Scandinavian  medical  students 
from  those  in  Canada  and  the  United  States,  but  not  the  United  Kingdom  or  Australia. 
Research  published  in  the  1980s  exhibited  the  highest  mean  effect  size  (ES  =  .84),  but 
was  limited  to  two  reports.  Most  research  was  reported  during  the  1990s  (63%  of 
studies),  but  mean  effect  size  estimates  were  statistically  similar  regardless  of  publication 
year. 

Substantive  features  not  moderating  effect  size  estimates  included  scope  of 
problem-based  instruction,  taxonomy  of  problem-based  instruction,  and  tutor  expertise. 
Seventy-one  percent  of  the  research  was  conducted  with  curriculum-wide  lecture-based 
versus  problem-based  instruction.  Although  mean  effect  size  was  greater  when  PBL  was 
implemented  for  one  course,  the  difference  was  not  significant.  As  with  other  outcomes 
(achievement,  problem-solving,  and  self-directed  learning)  examined  in  this  research, 
neither  reiterative  PBL,  nor  tutor  expertise  afforded  unique  superiority.   Substantive 
features  moderating  effect  size  estimates  included  institutions'  and  students'  experience 
with  PBL,  problem-type,  origin  of  learning  issues  responsibility  and  self-directed 
learning.  As  predicted  by  advocates  of  problem-based  teaching  and  learning,  ill- 
structured  problems  produced  the  greatest  effect  size,  but  this  difference  was  not 
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significant  for  PBL  formats  employing  structured  or  mixed  ill-structured  and  structured 
problems.  In  addition,  as  predicted  by  principles  of  situated  learning  (Brown  et  al.,  1989; 
Lave  &  Wenger,  1991 )  and  adult  learning  theory  (Kjiowles,  1984),  when  students 
exclusively  generate  learning  issues  and  direct  their  own  learning,  the  effect  of  problem- 
based  learning  is  better  realized;  although  these  differences  did  not  achieve  statistical 
significance.  There  was  significant  intercorrelation,  however,  with  effect  size  and 
learning  issue  responsibility  (r  =  .39,  p  =  .04).  Also,  significant  multicolinearity  among 
dimensions  is  evidenced  between  taxonomy  of  PBL  and  tutor  expertise  (r  =  .48,  g  =  .0 1 ); 
and  between  learning  issues  responsibility  and  self-directed  learning  (r  =  .4 1 ,  g  =  .03). 
Implication  of  the  latter  finding  is  that  creating  a  environment  in  which  students 
experience  ownership  of  learning  issues  can  facilitate  self  directed  learning, 

A  primary  purpose  of  curriculum  reform  is  to  enhance  attitudes  toward  learning. 
The  findings  of  this  research  suggest  that  there  are  differential  effects  of  problem-based 
and  lecture-based  instruction  regarding  student  perceptions  of  positive  attitude  toward 
learning,  utility  of  education,  and  level  of  academic  anxiety.  A  significantly  positive 
effect  of  problem-based  instruction  was  reported  in  57  percent  of  the  studies  regardless  of 
the  method  of  implementing  problem-based  teaching  and  learning  and  independent  of 
outcome  measure.  No  report  found  a  negative  effect  for  problem-based  teaching  and 
learning.  While  perceptions  of  learning  environment  are  important,  whether  they 
translate  to  competent  performance  cannot  be  examined  in  the  present  data  set. 

Summary.  Despite  diverse  methods  of  implementation  quantified  by  this  study, 
problem-based  instruction  tends  to  support  learning  conditions  informed  by  educational 
philosophy  (e.g.,  Deweyan  tradition)  and  the  principles  advocated  by  some  social 
construcfivists  (e.g.,  Cobb,  2002;  Prawat  &  Floden,  1994;  Savery  &  Duffy,  1995)  and 
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situated  learning  theorists  (Brown  et  al.,  1989;  Greeno,  1998;  Lave  &  Wenger,  1991) 
described  in  Chapter  2.  Functionally,  these  conditions  are  predicated  on  "locus  of 
control"  (Knowles,  1975),  whereby  learners  perceive  the  goals  of  the  learning  experience 
to  be  their  own  and,  as  a  result,  accept  a  share  of  the  responsibility  for  planning  the 
learning  experience  and  then  operating  in  it,  thus  becoming  actively  involved  and  vested 
in  the  process. 

A  Critique  of  the  Research 
Data  analyzed  in  this  research  only  are  as  valid  as  the  primary  studies. 
Consequently,  answers  to  questions  posed  by  this  research  are  represented  to  the  extent 
available  in  the  literature.  Ultimately,  however,  findings  and  inferences  resulting  from 
this  research  are  grounded  in  established  meta-analytic  methods  and  informed  by 
educational  philosophy  and  principles  of  cognitive  science  and  educational  psychology. 
Mam  effects  for  clinical  science  achievement,  problem-solving  success,  self-directed 
leammg  skills  and  positive  attitudes  toward  learning  are  robust  and  can  be  generalized 
with  significant  confidence,  but  analyses  of  some  moderators  were  limited  by  small  data 
sets.  Unfortunately,  details  of  implementation  are  missing  from  much  of  the  research 
literature,  which  reduced  the  sensitivity  of  analyses,  and  heterogeneity  within  categories 
of  many  study  features  could  not  be  resolved,  which  renders  unambiguous  inferences 
nnpossible.  Indeed,  a  significant  limitafion  of  the  available  comparative  research  is  that 
It  focuses  almost  exclusively  on  learning  outcomes.   Learning  strategies  and  cognitive 
processes  are  largely  ignored.  Such  a  research  program  is  incompatible  with  recent 
attention  afforded  the  principles  of  differential  psychology  and  social  construction  of 
knowledge,  which  were  discussed  in  Chapter  2.  Thus,  discovering  essential  elements  of 
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problem-based  teaching  and  learning  and  perfecting  them  remain  for  future  research 
guided  by  specific  gaps  identified  and  examined  in  the  present  research. 

Conclusions  and  Implications  for  Practice,  Researcli  and  Theory 

The  present  findings  suggest  that  responsibility  for  learning  can  be  transferred  to 
medical  students  resulting  in  biomedical  science  achievement  at  least  equivalent,  and 
clinical  science  and  problem  solving  achievement  superior  to  traditional  lecture-based 
approaches  while  fostering  a  positive  attitude  toward  learning  and  facilitating  greater 
self-directed  learning  in  students.  Although  effect  sizes  for  clinical  science  and  problem- 
solving  achievement  are  small  by  conventional  standards  (Cohen,  1988),  the  magnitude  is 
consistent  with  similar  instructional  interventions  in  college  students  (Lipsey  &  Wilson, 
1993).  Also,  considering  the  effect  size  in  present  findings  indicates  shifting  a  population 
of  medical  students  and  physicians,  "even  a  small  effect  size  is  equivalent  to  'moving  a 
mountain'"  (Albanese,  2000,  p.  730).  In  addition,  current  findings  suggest  that  the 
success  of  problem-based  teaching  and  learning  is  not  limited  to  a  specific  approach. 
Variance  analyses  revealed  diverse  operational  characteristics  and  variation  in  efficacious 
implementation  of  problem-based  teaching  and  learning.  Indeed,  analyses  revealed  that 
no  substantive  variable  of  problem-based  instrucfion  uniquely  predicted  any  cognitive 
outcome  effect  size.  However,  instances  of  multicolinearity  suggest  some  substantive 
features  may  converge  efficaciously,  as  a  function  of  subject  content,  context  and  desired 
cognUive  outcome.  In  addition,  despite  diverse  implementation,  small-group  problem- 
based  mstruction  was  conducted  ubiquitously  in  an  environment  fostering  the  integration 
of  biomedical  and  clinical  science  content  in  the  context  of  a  health  care  problem.  Since 
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lecture-based  instruction  also  can  integrate  biomedical  and  clinical  science  with  some 
evidence  of  achievement  parity  with  problem-based  formats,  the  significance  of  this 
finding  is  unclear  and  requires  further  research  to  examine  the  unique  variance  in  the 
achievement  effect  of  small-group  situated  versus  classroom  situated  learning 
environments. 

One  predicament  encountered  in  comparing  differential  effects  of  teaching  and 
learning  strategies  vis-a-vis  achievement  or  problem-solving  performance  is  that 
instruction  involves  varied  tactical  elements  and  apparent  success  or  failure  of  a 
particular  instructional  pattern  may  be  due  to  one  or  more  of  these  tactics,  rather  than 
overall  instructional  strategy.  Moreover,  because  all  teaching  and  learning  innovations 
react  and  adapt  pragmatically  to  their  particular  situation  (Cobb,  2002),  sustaining  an 
experimental  design  is  difficult,  thus  rendering  results  difficult  to  interpret. 
Consequently,  rather  than  investigating  global  instrucfional  effects  concerning  learning 
and  performance,  research  focused  on  specific  cognitive  aspects  of  differential 
perfomiance  may  be  more  explanatory,  particularly  since  conclusion  validity  is 
compromised  when  process  is  not  confirmed  (Messick,  1994).  From  this  perspecfive, 
research  would  be  more  appropriate  if  it  was  designed  to  examine  systematically 
cognitive  processes  that  guide  concept  formation  and  the  application  of  concept  in 
diverse  contexts  by  students  with  varied  aptitudes.  Research  aims  of  this  type  would  not 
be  to  compare  outcomes  of  instructional  format,  but  to  characterize  similarities  and 
differences  in  performance  that  may  arise  because  of  undergoing  training  in  one 
instructional  format  or  another.  For  example,  does  undergoing  lecture-based,  syllabus 
driven  instruction  (i.e.,  "other-directed  learning")  produce  clinicians  who  are  more  likely 
to  "react"  versus  "interact"  when  confronted  with  novel  clinical  problems?  Such  a 
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research  program  might  facihtate  unpacking  learning  processes  to  discover  efficacious 
vehicles  for  concept  formation.  This  could  be  a  rich  source  of  hypotheses  for  further 
investigation. 

Through  accumulated  research,  medical  educators  can  become  progressively 
more  competent  at  matching  students'  learning  predispositions  and  learning 
environments  to  maximize  their  educational  experiences  and  enhance  problem-solving 
skills.  Ultimately,  educational  research  strives  to  improve  the  quality  of  teaching  and 
learning.  Sustained  research  and  the  accumulation  of  results  are  required  to  gain  insight 
into  how  to  improve  medical  teaching  and  learning.  The  aim  is  to  take  advantage  of 
research  findings  and  convert  them  into  educational  interventions,  which  eventually  will 
improve  public  health  in  general,  and  individual  patient  care  in  particular. 

Present  findings  indicate  no  unique  superiority  for  either  lecture-based  or 
problem-based  instruction  for  affecting  biomedical  science  achievement.  Given  the 
current  explosion  of  biomedical  science  knowledge  (e.g.,  genefic  engineering,  stem-cell 
research,  recombinant  DNA  therapy,  etc.),  the  educafional  significance  of  this  finding  is 
paramount.  Framed  in  Dewey's  tradition  (1916,  1948,  1958,  1933/1998),  the  results  of 
the  present  research  not  only  argue  against  artificially  dichotomizing  the  instructional 
method  debate,  but  also  imply  the  potential  value  for  viewing  efficacious  medical 
pedagogy  as  a  dynamic  continuum  anchored  with  both  lectured-based  and  problem-based 
teaching  and  learning.  Indeed,  as  detailed  in  Chapter  I,  the  potential  strengths  of  pro 
PBL  arguments  resides  in  the  potential  weaknesses  of  con  PBL  arguments,  and  vice 
versa.  When  PBL  proponents  focus  intensely  on  the  attributes  of  student-directed 
activities  and  socially  constructed  knowledge,  the  more  they  expose  the  potential  for  a 
knowledge-base  deficiency.  When  PBL  opponents  focus  intensely  on  attributes  of 
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traditional,  lecture-based,  discipline-oriented  pedagogical  methods,  they  reveal  the  need 
to  connect  meaningfully  the  transmitted  information  with  clinical  practice  problems.. 

Evidence  revealed  by  the  present  research  suggests  curriculum  wide 
implementation  of  problem-based  instruction  is  not  required  to  realize  educational  aims. 
Further  investigation  is  warranted  to  examine  the  relative  merit  of  instructional  method 
versus  learning  responses  to  achieve  the  aims  of  medical  education.  As  suggested  by 
Dewey  (1929b,  1948,  1933/1998),  the  adaptation  of  instruction  whereby  interventional 
pedagogy  likely  is  best  when  it  is  evidence-based  and  appropriately  tailored  to  both  the 
aims  of  instruction  and  context.  By  this  means,  pedagogy  at  once  adapts  and  is  congruent 
with  the  educational  moment.  Indeed,  for  Dewey  education  was  an  experiment  (science 
of  pedagogy).  There  is  no  one  best  way  of  teaching  and  learning,  and  educational 
moments  would  drive  pedagogical  method. 

As  articulated  in  Chapter  2,  some  situational  learning  theorists  suggest  the 
essential  features  of  the  educational  moment  include  the  situational  awareness  and 
aptitude  of  students,  instructional  content  and  student-instruction  interaction  (Brown  et 
al.,  1989;  Lave  &  Wenger,  1991;  Savery  &  Duffy,  1995).  This  epistemological  stance 
considers  individual  differences  and  invites  the  emergence  of  teacher  talents  and 
creativity.  As  discussed  in  Chapter  2,  situational  aptitude  (e.g.,  content  expertise)  of  an 
instructor  also  may  play  a  role.  Indeed,  giving  explanations  may  help  instructors  better 
clarify  and  organize  their  knowledge  (Federman,  1999).  Although  potentially 
ambiguous,  a  classroom  as  laboratory  perspective  pemiits  the  infusion  of  new  ideas  from 
both  sides  of  the  lectern,  metaphorically  speaking.   Moreover,  a  pedagogical  method 
employing  controlled  or  planned  ambiguity  (e.g.,  cognitive  dissonance  or  indeterminate 
situation)  is  consistent  with  conditions  of  learning  espoused  by  Dewey  (1929b,  1948, 
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1933/1998),  and  with  some  views  of  social  constructivism  (Cobb,  2002;  Savery  & 
Duffy).  For  Dewey  (1933/1998),  "  Thinking  begins  in  what  may  fairly  enough  be  called 
a  forked-road  situation,  a  situation  that  is  ambiguous,  that  presents  a  dilemma,  that 
proposes  alternatives"  (p.  15). 

An  unequivocal  implication  of  this  research  is  that  teaching  and  learning  with 
versus  without  the  integration  of  biomedical  and  clinical  science  facilitates  superior 
clinical  science  achievement  and  problem-solving  skills  in  medical  students.  Therefore, 
dichotomizing  instructional  methods  and  research  focusing  outcomes  may  shed  more  heat 
than  light.  The  process  of  learning  must  be  systematically  examined  to  interpret  properly 
educational  outcomes,  account  for  instructional  impact,  and  thereby  link  the  learning 
process  to  educational  outcomes.  Substantial  work  remains  to  gain  an  understanding  of 
how  and  why  the  integration  of  biomedical  and  clinical  sciences  is  effective.  For 
example,  it  would  be  important  to  examine  the  differential  effects  of  instruction 
employing  the  integration  of  biomedical  and  clinical  science  knowledge  in  small  group- 
tutorials  compared  to  a  lecture-oriented  forniat.  Further,  more  research  is  essential  to 
develop  stronger  a  linkage  between  learning  theory  and  methods  of  medical  education. 
Investigations  designed  to  distill  this  process  can  infomi  practice,  research  and  theory. 

Summary.  Present  findings  suggest  that  there  is  an  instructional  difference  in  the 
method  of  medical  knowledge  acquisition  and  deployment  efficacy,  but  operational 
variability  in  instructional  methods  complicated  examination  of  moderating  features  of 
problem-based  teaching  and  learning.   Moreover,  significant  multicolinearity  of 
theoretically  grounded  substantive  features,  which  frequently  did  not  covary 
independently  with  outcome  effect  size,  prevented  the  isolation  of  unique  effect  size 
variance.  Despite  an  inability  to  identify  a  "silver  bullet,"  operational  variation  coupled 
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with  relatively  consistent  treatment  effects  across  studies  supports  the  efficacy  of 
problem-based  learning  as  a  legitimate  method  of  instruction  that  can  help  medical 
students  and  physicians  leam.  The  present  research  suggests  that  compared  to  lecture- 
based  instruction,  problem-based  teaching  and  learning  facilitates  equivalent  biomedical 
science  achievement,  but  mediates  significantly  better  clinical  science  achievement, 
problem-solving  achievement,  self-directed  learning,  and  positive  attitude  toward  medical 
education.  One  explanation  is  that  the  cumulative  effect  of  problem-based  teaching  and 
learning  is  greater  than  the  sum  of  its  parts.  Given  instances  of  multicolinearity,  quite 
possibly  substantive  elements  contribute  in  various  ways,  depending  on  content  and 
context.  Moreover,  since  problem-based  teaching  and  learning  variously  operationalizes 
principles  of  cognitive  psychology,  current  findings  inform  information-processing 
theory,  social  constructivism,  and  situated  cognition  in  general. 
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